Clouds

Magicjava Climate Data Announced

The first beta release of Magicjava Climate Data has been made available. The data is daily and monthly averages for AMSU channel 5 for January, 2009 through December, 2009 in .csv format. You can get the data here.

Aqua Satellite Project, Update 4 Released

Update 4 for the Aqua Satellite Project is ready. You can download it here. I'd have to say the big item in this release is the QA. But another big item is the new MapToGrid program.

MapToGrid

The MapToGrid program is for converting information in a black and white bitmap image to a 2.5 by 2.5 degree equal area grid that represents the Earth. An example image is shown to the left. It has water drawn in black and land draw in white. MapToGrid can take such information and transform it into C code, Javascript code, XML, or text. Other programs can then use this gridded information as a lookup table.

A screen shoot of a Javascript program that reads grid information and draws the Earth's water and land on an HTML page is shown below.

Javascript HTML image created using Draw2D and MapToGrid

MapToGrid is the first of several programs that, together, will allow the creation and display of gridded satellite data.

The command line help text for the MapToGrid program describes the details of using the program and is provided below.

MapToGrid converts a map image containing only the colors black and white into grid information. The map image must be a 32-bits-per-pixel, 720 by 360 pixel bitmap (.BMP) image. This image will be treated as a map of the entire world. The grid information will be a 2.5 by 2.5 degree equal area grid which indicates the percentage of white pixels in each grid.  The output grid can be formatted as C code, JavaScript code, XML, or text. The default is text. Usage:
MapToGrid [-h] -i Map_Bitmap -o Output_FileName [-f C | JavaScript | XML | text]
h Displays this help text.
i The location the bitmap (.BMP) input image.
o The name of the output file that will store the grid information.
f The output format. Valid format arguments are C, JavaScript, XML, or text.

There are sample bitmaps suitable for using with MapToGrid in the MapToGrid folder.

QA
A QA scrub was given to nearly all the existing programs (AMSUNormalize still needs to be done). Several major bugs were fixed and some new features added.

One of the new features is the the saving off of scans that don't pass NASA QA during an AMSUExtract run. You can take these rejected scans and run them through AMSUSummary just like the regular scans. A screen shot of a graph of rejected scans that has been summarized by AMSUSummary is shown below.

Another new feature is the -FLOOR switch added to AMSUExtract. This option lets you specify a lower value for scans. Any value below the floor is rejected. When using the -FLOOR switch you don't need to specify a value. In this case the value defaults to 150 K (-123.15° C, -189.67° F). The -FLOOR switch replaces the -q switch, which has been removed.

Yet another new feature added due to the QA was AMSUSummary printing out lower third and upper third averages for data rather than minimum and maximum values. This removes a lot of the noise that was created by the minimum and maximum feature. The maximum and minimum features have been removed.

A previous post showed a graph of the lower and upper averages that looked rather blocky. This has been fixed and you can use these features with full confidence.

Corrected Lower Third Averages Example

Change Log
So those are the big items. Here's the text of the change log for full details:


The following improvements were made for Update 4:

*) Added MapToGrid program that converts a map image containing only the colors black and
white into grid information.

The map image must be a 32-bits-per-pixel, 720 by 360 pixel bitmap (.BMP) image. This image
will be treated as a map of the entire world.

The grid information will be a 2.5 by 2.5 degree equal area grid which indicates the
percentage of white pixels in each grid.

The output grid can be formatted as C code, JavaScript code, XML, or text. The default is
text.

Includes command line help text.

*) Updated AMSUExtract program to send data that does not pass NASA's QA to stderr. Updated
amsu_extract script to handle this option. It sends stderr files to a file with the same
name as the regular file, but with the text "_Rejected" just before the file type suffix.

*) Added -FLOOR switch to AMSUExtract. The switch takes an optional value for degrees Kelvin.
Any scan below this value is sent to stderr rather than stdout. The value for -FLOOR defaults
to 150 K (-123.15° C, -189.67° F)

*) Removed -q switch from AMSUExtract. Limb adjustment checks are no longer done by this program.
NASA QA checks are done automatically now, as described above.

*) Default tolerance for Limb Check in AMSUQA raised from 0.5 to 1.0. Channels 12 and 19 added
to the list of channels that no limb check is performed on. Miscellaneous bugs in limb check
values fixed.

*) Modified AMSUExtract to report daily and monthly lower and upper 1/3 averages rather than
minimum and maximum values.

*) QAed AMSUExtract and AMSUSummary, fixing several major bugs.

*) MonthlyData and MonthlyDataMap have been refactored to a more generic ColumnData and
RowDataMap. These new classes are templates. The number of columns, ID type, and data type
are parameterized.

*) Added the ability to retrieve upper and lower averages to ColumnData and RowDataMap
classes.

*) Added avg and sum scripts. These scripts take a file name and a column number and return
the average or sum of that column. These are Unix scrips, requiring Unix utilities like cut,
grep, and awk. These scripts are not and will never be part of the main workflow, they are
intended as debugging aids. These new scripts are located in the Scripts folder. See the
comments inside the scripts for details on using them.

Results Of Aqua Satellite Project QA Checks

The big QA session for the Aqua Satellite project is done. The results are there were some major bugs in the code. With these bugs fixed, the data from NASA looks they way you'd expect it to. The unusual readings occurring for each footprint mentioned in this post are still there, but they're not enough to throw off the averages.

Major Bug Fixes
We'll take a look at all this data, but first, here's a list of the major bugs that were fixed:

● The Limb Check had values that were too small for some of the footprints in its lookup table. This is what was causing entire footprints to be dropped from the old QA checks.

● A subtle bug in the gnu C++ template generator was generating integers rather than floating point variables when working with floating point data. This was causing the decimal places to be chopped off, throwing off things like average values. This was fixed by getting rid of the offending templates and replacing them with a series of overloaded functions.

● The order in which I was grabbing data for daily and monthly summaries was wrong, causing data for one footprint to be displayed as another footprint. This has been fixed.

● The concept of daily and monthly minimum and maximum values produced by AMSUSummary was and is fundamentally flawed. For reasons why, see my post on Noise. Therefore, the concept has been replaced with averages for the lower third of the data and the upper third of the data. There's enough data that goes into these calculations that noise shouldn't be a problem.

What The Data Looks Like Now
The data displayed here covers the range of January, 2008, and January, 2009 through January, 2010. This is 14 months total of data. Letting the pictures speak for themselves:

Monthly Averages, By Footprint

Monthly Averages, By Date

Monthly Lower Third Averages, By Footprint

Monthly Lower Third Averages, By Date

Monthly Upper Third Averages, By Footprint

Monthly Upper Third Averages, By Date

Daily Averages, By Footprint

Daily Averages, By Date

Daily Lower Third Averages, By Footprint

Daily Lower Third Averages, By Date

Daily Upper Third Averages, By Footprint

Daily Upper Third Averages, By Date

The upper and lower values look a little blocky. So I'll do some more QA on those. I also still need to QA AMSUNormalize. However, none of those things are enough to prevent this code from being released as part of Update 4. Update 4 will be released later today.

Previous Posts In This Series
QAing AMSUExtract
What's Wrong With This Picture?

One Month Of AMSU Channel 5 Data

This video shows 1 month of Aqua AMSU Channel 5 data as it was displayed during one of my debugging sessions. The video is kinda blurry due to low resolution, but don't worry, you wouldn't be able to read all the data scrolling by even at HD resolution. Enjoy :)

Update:
Here's a screen shot of the end of the video in better resolution so you can see the data is nothing but the numeric readings from the channel.

National Snow And Ice Data Center Releases AMSR-E Satellite Surface Temperature Data

I just got this in the mail late yesterday. The National Snow And Ice Data Center (NSIDC) has just released several new climate products, including minimum and maximum air surface temperatures.

Here's the e-mail:

NSIDC is pleased to announce the release of the Daily Global Land Surface Parameters Derived from AMSR-E data set. This data set contains satellite-retrieved geophysical parameters generated from the Advanced Microwave Scanning Radiometer - Earth Observing System (AMSR-E) instrument on the National Aeronautics and Space Administration (NASA) Aqua satellite. Parameters include:

-Air temperature minima and maxima at approximately 2 meters in height

-Fractional cover of open water on land

-Vegetation canopy microwave transmittance

-Surface soil moisture at less than or equal to 2 centimeters soil depth

-Integrated water vapor content of the intervening atmosphere for the total column

The daily parameter retrievals extend from 19 June 2002 through 31 December 2008. The global retrievals were derived over land for non-precipitating, non-snow, and non-ice covered conditions. The primary input data were daily AMSR-E dual polarized multi-frequency, ascending and descending overpass brightness temperature data.

For more information regarding this data set, please see http://nsidc.org/data/nsidc-0451.html.

This is pretty timely. There's several simultaneous attempts going on right now to better understand Urban Heat Islands (UHI). The minima/maxima temperatures at 6 meters should be particularly relevant to these studies. A signature of UHI is the night time temperatures go up, while the day time temperatures stay about the same. This new data will give us the tools we need to look for UHI directly on a global scale.

And since the AMSR-E used to gather this data is part of the Aqua satellite, I can fold in studying the new data with the work on the satellite.

So expect to hear more about this in the near future.

QAing AMSUExtract

In A NutShell
So, I've been QAing AMSUExtract the last couple days. The end results is that several bugs were found and fixed in my code. But even with these fixes, a lot of data was still failing limb validation. Even after increasing the limb validation tolerance from the original 0.5 to 10.0 (giving over 100 degrees tolerance in some cases), there were still more failures than I would consider acceptable.

This told me that having a limb validation in AMSUExtract was pointless. To get a decent number of records to pass this QA check, I'd need to make the check so loose that nearly any imaginable value would pass.

So limb validation, the -q switch of AMSUExtract, has been taken out. It's been replaced with a -FLOOR switch that lets you set a lower bound for a valid temperature. The default lower bound when you use -FLOOR is 150 degrees K (-123.15° C, -189.67° F), but you can pass any numeric value you want. Scans that are marked as bad by NASA or that fail the optional -FLOOR check are the only ones that will now fail in AMSUExtract.

Limb vaidation is still available as an option for the AMSUQA program.

I want to thank Malaga View for suggesting the feature to save not only scans that pass QA, but also scans that fail QA. This feature is now part of AMSUExtract and was very useful for this QA check. BTW, should anyone else have ideas for features, please feel free to suggest them. If I have the time, I'll add them.

The end result of all this is AMSUExtract is now working and has new features to boot. This will be released in Update 4. Now I move on to QAing AMSUSummary.

Final Results With -FLOOR switch. 

The file is clean like this all the way through.

The Painful Details

The part of this post can be skipped. It's the details on how the QA was carried out. It's here for people who just love to read about QA procedures on blogs or who may need to know the details for something they're doing on their own.

Setup For QA

I setup the QA runs by removing all but one folder for the directory I was working in. That folder contained .hdf files for a single month. That folder, in turn, contained a text folder that contained the text extract files created by ncdump from the .hdf files.

As always, ncdump was run with no command line arguments, just the file name of the file being converted.

You can download ncdump from here.

Running Release 3 Version Of AMSUExtract

The amsu_extract script was used to extract data from all the files for the month. The exact command line was:

amsu_extract -extension C5F1-30_QA_TEST -c 5 -f 1-30 -q

This extracts channel 5, footprints 1 through 30 and performs QA checks on them. The -extension argument tells the script how to name the output file. In this case the output extract file will be named 200912_extract_C5F1-30_QA_TEST.csv, where 200912 is the name of the folder storing the monthly data. This file is created in the same directory from which the script is run.

The version of AMSUExtract called by the amsu_extract script was the same as the one in Update 3 with one exception: scans that fail QA are sent to stderr. The amsu_extract script, in turn, saves stderr to a separate file.

Results Of Running Release 3 Version Of AMSUExtract

Step 1: Check Values Being Extracted

The first thing to do is verify that AMSUExtract is pulling the correct data. To do this, use HDFView and open up the original .hdf files downloaded from NASA. You can download HDFView from here.

Use the File/Open command to open the .hdf files you want to look at. Use the tree control on the left to navigate to brightness_temp. Double click on brightness_temp. At the to of the grid window that opens up, use the page scroller to move to page 4. Page 4 is the data for channel 5, the numbering starts at zero. 

You'll notice the rows in the window are numbered 0-44. These are the scan lines. The columns are numbered 0-29. These are the footprints. 

Open up the extract file (200912_extract_C5F1-30_QA_TEST.csv in my case) in a text editor. Visually compare the values in the extract file to the values in HDFView. You don't have to check every value, but you want to check enough cases to convince yourself the correct values are being pulled.

Step 2: Compare File Sizes Of Passed And Failed Data Files

Checking the file sizes of the passed and failed data shows that the failed file is about 2/3s the size of the passed file. Now, a fair portion of that is header information, but most of it is actually failed data. This means a lot of data is failing QA.

Step 3: Look At Failed Data:

Opening up the failed data file, I saw a lot of data that looked reasonable to me. So I was failing data that should have passed. 

To fix this problem, I tracked down a couple of bugs in the QA checks, expanded the number of footprints that don't get a limb validation check, and increased the tolerance of the limb validation from 0.5 to 10.0.

Even with these changes, a lot of data was still failing. I'll talk about that in the next section.

Results Of Running Modified Version Of AMSUExtract

To try to cut down on the number of scans failing, I gradually increased the tolerance. Eventually, I had it all the way up to 10.0 and still had more records failing than I expected. Screen shots of running with a 10.0 tolerance are below.

The error file looks a lot better, but further down the file there are still quite a few scans listed as failing.

At this point I realized it was hopeless to try and salvage the concept of running a limb validation in the extract. So I took it out and replaced it with the -FLOOR option.

References:
Download ncdump
Download HDFView