Looking At The Aqua Satellite Data

In this post we're going to take a look at the Aqua satellite data we've downloaded. And that's all we're going to do: look at it. We're not going to perform calculations with it or even try to understand it. We're just going to take a look at it to get some awareness of the what data is there, not what the data actually means.

There are 3 tools we're going to use to look at the data: 1) ncdump, 2) vshow, and 3) HDFView.

If you haven't already downloaded the HDF-EOS tools, you can get them here. The ncdump and vshow tools are located in the "utilities" folder of the download.

You can download HDFView from here. HDFView is a Java program. Unzip and double click the installer to install the program on your computer.

If you haven't read the previous posts in this series, they're located here:
UAH Satellite Data
Dangit! More Climate Stuff. UAH and RSS Raw Data

ncdump
The ncdump tool is a command line utility that converts an hdf file to ASCII text. It has several options, the first of which you should learn is the -H (upper case H) option. This is the help option that gives you information on all the other options available. These options are:

./ncdump [-c|-h|-u] [-v ...] [[-b|-f] [c|f]] [-l len] [-n name] [-d n[,n]] file
[-c] Coordinate variable data and header information
[-h] Header information only, no data
[-u] Replace nonalpha-numerics in names with underscores
[-v var1[,...]] Data for variable(s) ,... only
[-b [c|f]] Brief annotations for C or Fortran indices in data
[-f [c|f]] Full annotations for C or Fortran indices in data
[-l len] Line length maximum in data section (default 80)
[-n name] Name for netCDF (default derived from file name)
[-d n[,n]] Approximate floating-point values with less precision
file File name of input netCDF file

The -h (lowercase h) option will extract only header information. So this tells us an HDF file has headers and gives us a way to look at what those headers are. I ran ncdump using this option from a terminal window, like so: ./ncdump -h filename | more. A screenshot of the output is shown below.



We can see there are headers with names like DataTrack_lo:Low_Res_Swath, Antenna_Coeff:Low_Res_Swath, DataTrack_lo:High_Res_B_Swath, Latitude(DataTrack_lo:Low_Res_Swath, DataXTrack_lo:Low_Res_Swath), and Longitude(DataTrack_lo:Low_Res_Swath, DataXTrack_lo:Low_Res_Swath).

I'd encourage you to run ncdump like this at least once for UAH data and once for RSS data and take the time to scan through the results.

The next option we want to look at is -b. This provides a brief description for data items that are part of indexes (arrays in C-speak). You need to give this option an f or c argument. This indicates whether the data should be presented in a Fortran-like or C-like format. I prefer C over Fortran, so I used this command: ./ncdump -b c filename | more. The first part of the results look a lot like the header results. But scrolling further through the file reveals differences. A screenshot is shown below.


You can see we're now getting values for the headers we saw previously.

The final ncdump command we're going to cover is -f. This is like -b, but gives full descriptions for the data items. Using the command ./ncdump -f c filename | more produces the following results:


As you can see, every item in an array is commented with its exact location in the array. I have a feeling this kind of output may come in handy during debugging situations.

So the four variants of ncdump we've discussed are:
./ncdump -H
./ncdump -h filename | more
./ncdump -b c filename | more
./ncdump -f c filename | more

These commands will show the Help screen, print out headers, print out data with brief descriptions and print out data with full descriptions.

If you want to save the output to disk, replace | more with > output_filename

vshow
The vshow command produces output similar to ncdump when used with the -h option. That is, vshow produces a list of the headers in the file. However, vshow goes a bit further and shows the sub headers under each header. The output of the command ./vshow filename | more is shown below.


Vshow calls the headers v groups and labels each header as vg #, where # is a sequential number starting with zero. The sub-headers are labeled the same way, with each sub-header number also starting at zero.

HDFView
The final tool we're going to look at is HDFView. This tool provides a graphical interface to HDF files. It shows both headers and data. You can edit the file and save your edits to disk.

To use this tool, double click on it, and select Open from the File menu. Navigate to the file you want to look at and click the Open button. A screen shot of HDFView with a file loaded up is shown below.


On the left side you see a Tree control showing the headers in the file. Double-clicking a header brings up it's edit window in the main window.

If you make edits to the data they can be saved using the Save or Save As menu items in the File menu.

References:
NSIDC HDF-EOS To ASCII Tools
HDFView
UAH Satellite Data
Dangit! More Climate Stuff. UAH and RSS Raw Data

UAH Satellite Data

I made a mistake about the UAH data in my previous post. I had assumed UAH and RSS data had the same data source. This isn't true. So this post will cover how to download the raw UAH data.

The good news is the data for AMSU is stored in the same format as the data for AMSR-E that we discussed previously. The bad news is getting this data from NASA is more complicated than getting data from NSIDC.

Getting UAH Aqua Satellite Data
The AMSU data used by UAH is stored by NASA at their AIRS data holdings web page. Follow these steps to get the data:


On the DATA PRODUCTS Screen you'll see a table at the bottom of the screen. On the left-most column you'll see a link that provides the general data product name. The second column describes a subset of the general product data, and the right most column has a link to where the data can be retrieved.

The data we want is the AMSU-A1 & AMSU-A2 combined, geolocated & calibrated brightness temperatures, which is in the 3rd row of the table. To start the process of downloading this data, click the Mirador link in the right-most column of the 3rd row.


The second screen lets us specify location and time criteria for the data. In this example, I'm entering a date range of 01/01/2010 (in MM/DD/YYY format) for the beginning and end ranges. I'm leaving the location criteria blank so that I get data for the entire planet.

Once the search criteria is entered, click the Search GES-DISC button on the form.


The next screen shows the results of the search. This screen is deceptively simple, showing what appears to be only one file. This isn't actually the case. There's actually dozens of files. Where the NSIDC RSS files each contained about 50 minutes of data, the NASA UAH files contain only 6 minutes of data. So we're going to get lots of files to cover just one days worth of data.

To see all the files actually in the results, click the View Files link below the result.


This screen lets you pick and choose which files that were part of the search results that you actually want to download. There's two useful buttons on this page, the Add Selected Files To Cart and the Add All Files in All Pages To Cart buttons. This form uses a shopping cart metaphor . You add the files you want to your shopping cart, then download the contents of your shopping cart.

Since this is just a walkthrough, I'm only going to select one file, the first in the list, for download and ignore the others. This will give me 6 minutes of data for the date I selected. I do this by unselecting all the files except the first and clicking the Add Selected Files To Cart button.


Now we're at the Services Selection screen. Here we can select how we want to download the files (by HTTP or FTP), or further narrow the results to provide only a subset of the channels in the data. Different channels measure different parts of the atmosphere.

I'm going to just grab all the channels, so I can ignore the channel service. And for this particular search result I only have the option of using HTTP as the download protocol, so that's what I'll use. I do this by clicking the Download via HTTP link on the right side of the screen. This puts the file in my shopping cart.

Now I click the Continue Shopping button.


I'm now at a screen where I can choose to do another search, or check out. I've gotten all the data I want, so I click the Check Out button.


This is where it gets complicated, so pay close attention.

The NASA server doesn't actually download files to your computer. Instead, you have to send a series of commands to the NASA server telling it to give you the files you want. Using the following screens you'll build those commands, copy them to your computer, and then execute them to get the files in your shopping cart.

The screen shown above provides three different methods to issue commands to the NASA server to download your results. These methods are 1) A series of wget commands, 2) a series of curl commands, or 3) downloading a Java client to your machine and using that to issue the commands.

Non-technical users will want to use the Java client and follow any instructions it provides. To do that, click the More Options... link at the bottom of the page and follow the instructions provided. The rest of this post will discuss using curl to download the files.

The wget and curl methods require a file containing the list of files to be retrieved, and command line instructions needed to download the files in the list. The command lines for wget and curl can be seen in the screen shot above. You'll want to copy and paste one of those command lines after you have the file list on your computer.

My machine is a Macintosh OSX box. This is a UNIX box that has curl, but not wget, so I'll use curl. Windows users will want to install wget or curl on their machine and use that.

Notice the three buttons at the top of the form, URL List (Data), URL List (Metadata), and URL List (Data and Metadata). Clicking one of these buttons will open a new window containing the file list needed to get the files.

I want data and metadata, so I click that button. The resulting screen is shown below.


This screen shows the file list that need to be used to download the data. Copy and paste this list into a text file. On my Macintosh I copied the file list, opened a terminal window, started a text editing program called vi, and pasted them into vi. Then I saved the file to my hard drive with the name myfile.dat. Exiting vi, I used the chmod command to give execute permissions to the file.

Now close the browser window containing the file list and go back to the window containing the wget and curl instructions. Copy and paste them to the command line to execute them. This will download the files from your cart to your computer.

A screen shot of the terminal session I used to do all this is shown below (with my real name covered up).



Wrapping Up
The NASA method of getting the data is much more user-hostile than the NSIDC method. There's nothing I can do about that. But you now know how to get the UAH raw data.

References:
Downloading and overview of RSS Satellite data
NASA AIRS data holdings web page

Dangit! More Climate Stuff. UAH and RSS Raw Data

UPDATE:
It turns out the data described in this post is used to determine ice and other terrestrial, atmospheric, and oceanic variables, not RSS data. To the best of my knowledge, raw RSS data isn't available online. See this post for more information.
=================

Well, so much for getting back to my physics code. There seems to be a hole in the sceptic community's understanding of raw UAH temperature data. I really think this hole needs to be filled and I think I'm going to have to be the one to fill it. I'm going to post the steps I'm taking along the way. This way there's a written trail for others in the community to follow if they want to. It'll also give anyone out there who's familiar with raw UAH temperature data the chance to tell me if I'm going off into the weeds, or if there's a better way to do what I'm doing. I'm hardly a "satellite raw data expert", so if someone out there sees me making a mistake, don't be shy pointing it out. :)

This post describes how to get the raw data, what format that data is in, and how to get a few utilities for working with the data. So, here we go...

UAH And RSS Data
First of all, what is UAH and RSS data? They're both satellite data, one comes from the University of Alabama, Huntsville (UAH) and the other comes from a company called Remote Sensing Systems (RSS). Both of these data sets are processed data derived from the same raw source. This source is NASA's Aqua satellite. You can read about this satellite and how UAH uses its data in this article at the Watts Up With That Blog.

So if we want the raw data for UAH and RSS, we want the data from the Aqua Satellite.

Getting Aqua Satellite Data
After doing some searching on the net, I found the raw data for the Aqua satellite at the National Snow and Ice Data Center (NSIDC) Order Form. Follow these steps to get the data you're looking for:


On the Order Data Screen, click the Data Pool link.


On the DataPool screen, click the AE_L2A.2 link. This contains the AMSR-E/Aqua global swath brightness temperatures data.


Now you're at the data selection screens. This is a group of screens that let you select the date and time of the data you want, as well as spatial and day/night data. NOTE: Daily data can range in size from 1 to 2.5 GB! So you're not going to want to download tons of data at a time. Stick to small date ranges, probably one or two days.

In this example, I'm interested in the last full day of data, which is currently 12 JAN 2010. At the calendar at the bottom of the screen, I click that date.


The next screen let's me pick the time of day I'm interested in. I'm interested in the entire day and I don't want to set any day/night or spatial options. So I'm ready to go and can move on to the download screen. I do this by clicking the Get the granules that match the above criteria link in the middle of the screen.


Now I'm at the screen showing the results of the search. At the bottom of this screen is a list of "granules" that matched my search. These granules are the data files to be downloaded. If you look at the top of the Results box at the bottom of the screen, you'll see that the first 10 granules out of 30 granules are being displayed. You'll also see controls for displaying the other granules. There are 30 granules in all and we want all 30.

We start downloading by right-clicking the link to each granule and selecting Download. You can see the links to the granules on the left side of the Results box at the bottom of the screen. The first link in this example is named SC:AE_L2A.2:35160180. There's a row for each granule. Download 10, then go to the 2nd screen, download 10 more, then go to the 3rd screen and download the last 10.

Congratulations, you now have all the raw Aqua data for Jan 12, 2010.

Aqua Satellite Data Format
The Aqua Satellite data files are in a format known as Hierarchical Data Format - Earth Observing System (HDF-EOS). These files contain a mix of binary and text data. The text data is in a hierarchical format that is custom designed by NASA. There's a page at the NSIDC web site that describes this data. It says:

Hierarchical Data Format (HDF) is the standard data format for all NASA Earth Observing System (EOS) data products. HDF is a multi-object file format developed by The HDF Group.
The HDF Group developed HDF to assist users in the transfer and manipulation of scientific data across diverse operating systems and computer platforms, using FORTRAN and C calling interfaces and utilities. HDF supports a variety of data types: n-Dimensional scientific data arrays, tables, text annotations, several types of raster images and their associated color palettes, and metadata. The HDF library contains interfaces for storing and retrieving these data types in either compressed or uncompressed formats.
For each data object in an HDF file, predefined tags identify the type, amount, and dimensions of the data; and the file location of various objects. The self-describing capability of HDF files helps users to fully understand the file's structure and contents from the information stored in the file itself. A program interprets and identifies tag types in an HDF file and processes the corresponding data. A single HDF file can also accommodate different data types, such as symbolic, numerical, and graphical data; however, raster images and multidimensional arrays are often not geolocated. Because many earth science data structures need to be geolocated, The HDF Group developed the HDF-EOS format with additional conventions and data types for HDF files.
HDF-EOS supports three geospatial data types: grid, point, and swath, providing uniform access to diverse data types in a geospatial context. The HDF-EOS software library allows a user to query or subset the contents of a file by earth coordinates and time if there is a spatial dimension in the data. Tools that process standard HDF files also read HDF-EOS files; however, standard HDF library calls cannot access geolocation data, time data, and product metadata as easily as with HDF-EOS library calls.

From a computer programmer's point of view, we've got a custom data format that is extremely flexible but is going to be a bit of a pain to parse. Luckily, NASA provides some tools to make it easier to work with the data.

Tools
There's a page at the NSIDC web site that provides links to a few HDF-EOS tools. One of these tools converts HDF-EOS files to ASCII. I believe this is the tool we're going to want to use.

The tool is called ncdump, and it has its own web page. On that page you'll find instructions for installing and using ncdump on UNIX and Windows platforms. I'm not going to repeat those instructions here. Just follow them and you'll have ncdump as well as several other tools installed on your system, ready to use.

Note that if you're asked for a user ID and password for FTPing the files to your computer, log on to FTP as an anonymous user (Guest) and no password.

Wrapping Up
That's all I'm going to cover for this post. At this point you know about UAH and RSS raw data from the Aqua Satellite. You know where to get that data and how to download it to your computer. And you have a few tools for working with the data.

This is as far as I've gone with the process so far. There's a lot here and I wanted to stop and document the steps I've taken before moving on.

References:
How the UAH Global Temperatures Are Produced
National Snow and Ice Data Center AMSR-E Aqua Data Order Form
NSIDC Introduction to HDF-EOS
NSIDC HDF-EOS tools
NSIDC ncdump Tool Web Page

Getting Back To Normal

I think at this point I've gotten out all the information I've wanted to on the climate in general and ClimateGate in particular. That means I'll be going back to the physics code I was doing before ClimateGate broke. I want to thank all the folks who've been reading, who downloaded the Climate Scientist Starter Kit, and especially the 50,000+ people who watched my ClimateGate Who's Who video on YouTube.

I'll probably be getting back to the climate this spring when the EPA tries to push cap and trade. Till then, I need to concentrate on my physics code.

Thanks again everyone.

Climate Scientist Starter Kit, Version 1.5

The Climate Scientist Starter Kit has been updated to version 1.5. There's lots of new data and this video explains what you'll find.

This video has been placed in the public domain.

Climate Scientist Starter Kit Simple Data Analysis

This video shows you how to do some simple data analysis with the Climate Scientist Starter Kit spreadsheet. It includes an example of re-creating Svensmark's cosmic ray and tropospheric temperature graph.

This video has been placed in the public domain.

Climate Scientist Starter Kit Introduction Video

This video is a walkthrough and explanation of the data found in the Climate Scientist Starter Kit spreadsheet.

The video has been placed in the public domain.