Aqua Satellite Project, Update 1 Released

After a hard weekend of coding, Update 1 for the Aqua Satellite Project is ready. You can download it here. It includes a new AMSUExtract program for pulling temperature data.

Future posts will take a look at the functionality provided by the update, but for now here's the change log entry on what's been added:


=== UPDATE 1 February, 28, 2010
The following improvements were made for Update 1:


*) Speed improvements. The HDFReader class got a significant boost in speed by adding internal indexing for all variable access.


*) AMSUQA output modified to csv format for easy viewing in spreadsheets.


*) Help text added to AMSUQA command line options.


*) Added AMSUExtract program for extracting temperature data in csv format. Includes command line help text.


*) Code cleanup. Helper functions such as split() where previously repeated in every .cpp file that needed them. The standard method of putting such functions in headers is now used. The placement of code in folders has been cleaned up as well.


*) HDFNode and HDFNodeDocument have been refactored to a more generic Node and NodeDocument. These new classes are templates. The data type and attribute data type are parameterized. Two typedefs are provided for string data and attributes. These typedefs are named StringNode and StringNodeDocument.


*) NodeDocument query syntax has been modified to make it more in line with XPath syntax. A separator character must now appear at the beginning of a query string. All existing code that was using queries has been updated to match the new syntax. 


*) Bash shell script for running AMSUQA has been added. It's name is amsu_qa and it's located in the Scripts folder. This script will convert compressed binary HDF files to text as needed. See the comments inside the script for details on using it.


*) Bash shell script for running AMSUExtract has been added. It's name is amsu_extract and it's located in the Scripts folder. This script will convert compressed binary HDF files to text as needed. See the comments inside the script for details on using it.


*) Bash shell script for converting compressed binary HDF files to text has been added. It's name is hdf_to_text and it's located in the Scripts folder.  See the comments inside the script for details on using it.


*) Macintosh XCode project for AMSUQA and AMSUExtract added. The build folders of these projects includes binaries that will run on a 64-bit enabled intel Macintosh.


*) Added this change log.

Previous Articles In This Series:
Spot Checking The Spot Check
NASA, UAH Notified Of QA Spot Check Findings
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

References:
Aqua Satellite Project (BETA) Update 1

Spot Checking The Spot Check

Before I posted the results of the QA spot check described here, I first checked those results to make sure they were accurate. This post describes how I checked them.

Checking Binary Data

The Aqua Satellite AMSU Level 1B data comes in compressed binary form. You can view the data using HDFView, which is a tool created by the HDF Group. I used this tool to go in to the data files exactly as they were downloaded from NASA and look at the data. The screen shot to the left shows the values of Channel 4 QA that I reported. Any value other than zero means the channel failed QA. As you can see, all of Channel 4's readings failed. Channel 4 is labeled as "3" because the numbering in the viewer starts at zero rather than one.

Checking Text Data
HDFView is an HDF viewer, not an HDF-EOS viewer. And the Aqua AMSU Level 1B files are HDF-EOS files. This means HDFView can show you the HDF parts of the file, but not the EOS parts of the file. To look at the EOS data in the file, I converted it to text and opened it up in a text editor. Conversion to text was done by taking the original NASA data file and running it through ncdump with no arguments other than the file name to be converted.

The two screen shots below show samples of EOS data for two different files. You can see the file names as the titles to the window. Looking at the highlighted data, you can see the name of the data shown and its value. The data is NumBadData and the value for both files is 1350.

These types of checks were done on about 20 randomly selected files to ensure the errors being reported by the QA code were actually due to the data in the files, not due to bugs in the QA code itself.

Previous Articles In This Series:
NASA, UAH Notified Of QA Spot Check Findings
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

References:
UAH January Raw Data Spot Check
Looking At The Aqua Satellite Data

NASA, UAH Notified Of QA Spot Check Findings

I've sent off the results of my QA Spot Check to the Aqua team at NASA and the Dr. Christy and Dr. Spencer at UAH. I couldn't find anything wrong with my results, so it's time to let the professionals have their input.

Here's a copy of the text that was sent:

I'm writing with questions regarding the January, 2010 Aqua satellite AMSU Level 1B data. I checked various QA flags in the data and found the following results:

● Of the 7410 files containing January data, 7386 of them had their automaticQualityFlag marked "suspect" and another 24 of them had the flag marked "failed". None passed QA.
● Of the 7386 suspect files, all of them had a "Good Data" percentage of 93.33334 percent. Here, "Good Data" is taken as the result of subtracting bad data, special data, and missing data from total data. "Good Data" is then divided by total data to get the percentage of "Good Data".
● Of the 333,450 Channel 4 readings for January none of them passed QA. All of them in the files marked "suspect" had been marked as failing QA and, obviously, the ones in the files marked "failed" were in files that failed QA and should not be used.

My questions are:

● Is it considered normal to have zero Level 1B AMSU data files for a month pass QA?
● Is it normal for all Level 1B AMSU data files for a month to have the exact same numbers for bad data, missing data, special data, and total data?
● Doesn't the statistics engine used for AMSU limb adjustment require valid data from channel 4 in order to correctly adjust channel 5 data?

Additionally, I asked Dr. Christy and Dr. Spencer if channel 5 from Aqua's AMSU is used to produce UAH anomalies. In an article on WUWT, Dr. Spencer said it is, but I just wanted to double check to make sure I understood correctly.

Previous Articles In This Series:
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

References:
How the UAH Global Temperatures Are Produced

About The Aqua Satellite Project

Folks are downloading the code from the Aqua Satellite Project, so I wanted to do a short post about the project.

Purpose
The purpose of the project is to provide a set of tools for accessing and using data from the Aqua satellite. The goal is to be able to replicate as much of the processing done to the data as allowed by law.

State
The project is in its very early stages. More will be added over time, and if you find any mistakes in what's there, please let me know and I'll correct it.

Aqua Satellite Book

I've been working on a book about the Aqua satellite as part of the project. This book isn't a coffee table book or an "executive summary". It digs into the details of how the Aqua satellite works. Because people are downloading the Aqua satellite code, I've added a "early access" version of the book to the download section of the blog under the name Aqua Satellite (PDF). So far, the book is over 50 pages long and there's more to come. Use this book to come up to speed on the technical aspects of the satellite and its related data and processing code.

Short Source Code Overview
I'll close out by giving a brief overview of the code I used to do the QA spot check from this post. Please remember that this is early beta code.

HDFReader.h and HDFReader.cpp - These files contain the code to read a generic HDF-EOS file. The code works, but requires an ASCII version of the HDF-EOS file rather than the binary files that come from NASA. Use the utility ncdump with only the filename as an argument to convert NASA's binary files to text. The Aqua Satellite book has instructions on how to get ncdump.

AMSUReader.h and AMSUReader.cpp - These files are specializations of HDFReader designed to make it easier to read AMSU HDF-EOS files.

CommandLineArgs.h and CommandLineArgs.cpp - Defines a C++ class for working with command line arguments passed to a program.

FileUtils.h and FileUtils.cpp - Provides some cross-platform file utilities.

main.cpp - The entry point to the QA spot check code I wrote.

AMSUQA.h and AMSUQA.cpp This file  contains the actual QA spot check code. It's translated to pseudo-code below.

   Load AMSU Data

   // Check QA
   Set NumberOfGoodData Equal To NumberOfData - NumberOfBadData - NumberOfMissingData - NumberOfSpecialData
   Set PercentGoodData Equal To NumberOfGoodData Divided By NumberOfData
   Multiply PercentGoodData By 100

   Foreach scanline
      Foreach channel
         If channel Failed NASA QA, Add 1 To Channel's Bad Count
      End Foreach
   End Foreach

   // Write QA results
   Open Output File Using Value Of AutomaticQualityFlag As Filename
   Write Out Name Of AMSU Input File
   Write Out Percent Good Data
   Foreach Channel
      Write Out Channel's Bad Count
   End Foreach

   Close Output File

References:
UAH January Raw Data Spot Check

UAH January Raw Data Spot Check

Update 2:
The C++ source code used to run these tests has been added to the Downloads section of the blog under the name Aqua Satellite Project (BETA).

Update:
While I said words to this effect at the end of the post, I wanted to make sure it's clear: These findings are preliminary. Although nearly two months of research has gone into them more needs to be done.

In the meantime, folks wanting to see the data for themselves can download it from NASA by following the steps here: UAH Satellite Data and downloading the viewers by following the steps here: Looking At The Aqua Satellite Data.
======

Killing two birds with one stone, I decided to give my AMSU C++ code a bit of a workout by doing a Quality Control spot check on the January UAH raw data. I downloaded all the January Level 1B data for the AMSU from NASA. There are 7410 files for the month of January. Here's what I found:

No Files Passed NASA's QA. None.
Of these 7410 files, 7386 of them had their automaticQualityFlag marked "suspect" and another 24 of them had the flag marked "failed". Add that up and it's all 7410 files for January were either suspect or failed. No a single file was marked as "passed".

Of the 7386 suspect files, all of them had a "Good Data" percentage of 93.33334 percent, meaning that's the percentage of data in each file that's considered reliable. There's no way such a number would actually be identical out to one one hundred thousandths of a percentage 7386 times out of 7386 tries, so I figured my code that was reading the wrong data for the good/bad/missing/special values.

Nope, my code was good and manually checking a random selection of about 20 files by opening them up in an editor revealed they all had the identical numbers in them for these fields. In other words, the good/bad/missing/special data flags seem to have hard-coded values, at least for "suspect" data.

Channel 4 Never Works (Which Means Channel 5 Never Works Either)
Each of the the files contains 45 readings from each channel. Of these 45 readings, channel 4 failed 45 times every time. The only exception was in the files that had been marked as "failed". So there was never a good reading from channel 4 for the entire month of January.

This is significant because the statistics engine used to correct the satellite readings for channel 5, the channel UAH uses to get it's anomaly information, requires data from channels 4 and 6.

So this means they're either using a non-published algorithm to calculate the anomalies, or the anomalies are wrong.

Screen Shoot Of QA Spot Check Results

(Click for larger image)

Disclaimer
Here are a few things I think of that may affect the validity of this spot check:

• I'm not a scientist, I'm a software engineer. 
• Prior to working on the Aqua satellite as documented in this blog, I had no experience on working with satellite data. 
• Despite having checked my code, it is beta code, and I may have missed errors.
• It's possible that data files are not given "passed" status (as opposed to "suspect" and "failed") until some QA procedure that takes place after Level 1B occurs.
• It's possible the order of channel information in the data file does not represent the actual channel number. In other words, the fourth channel in the data file may not be channel 4.
• UAH uses other satellites in addition to the Aqua AMSU.

While I think it's important to point out these possible concerns, I also think it's important to point out I don't see any of them as serious threats to reversing what I'm seeing in the data now.

Still, further research needs to be done to triple check these things, and that's what I'm working on now. If I don't come across anything that changes my views, I'll forward these results to Dr. Spencer and Dr. Christy for their input.

Conclusion
At this point, while admitting the possibility of an error on my part, I'd have to say it seems to me the January AMSU data has no validity at all. Nor does anything derived from it. The hard-coded quality values and the complete failure of channel 4 both look like show-stoppers to me.

Previous Posts In this Series
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

References:
The Limb Adjustment of AMSU-A Observations: Methodology and Validation

Calculating Integrals In The Climate Scientist Starter Kit

Since I showed you how to calculate the derivatives of CO2, I figured I'd show you how to calculate the integral as well. Just like calculating derivatives, calculating integrals is surprisingly easy.

Calculating Integrals In The Climate Scientist Starter Kit
The first thing we need to know is just what an integral is. Another name for integral is "area under the curve", and that's just what an integral is, the area from the bottom of the graph to the place where the lines are. To demonstrate this visually, I plotted a graph that has CO2 twice, once as a line and once as a set of bars. The distance between the bars was set to zero. Here's what the graph looks like:

You can see that when CO2 is plotted as a set of bars, it completely fills up the area under the curve of CO2 plotted as a line. This is exactly what an integral is! So we know there's enough information just from the CO2 values to get the integral. So how do we get the actual number value for the integral?

It's simple. We add up the values for all the bars. This can be done with a simple sum() function:

=SUM(COLUMN_NAME)

Done!

Calculating Integral Ranges
Suppose we want to find the integral of just part of the graph, say between the 5th and 10th months listed. This can be done by taking the integral for each range and subtracting the results.

=SUM(COLUMN_NAME_ROW_1:COLUMN_NAME_ROW_OF_FIRST_VALUE)
=SUM(COLUMN_NAME_ROW_1:COLUMN_NAME_ROW_OF_SECOND_VALUE)
=SECOND_INTEGRAL - FIRST_INTEGRAL

Example:
=SUM(C1:C5)
=SUM(C1:C10)
=SECOND_INTEGRAL - FIRST_INTEGRAL


The first function gives us the integral of columns 1 through 5. The second function gives us the integral of columns 1 through 10. Subtracting the first integral from the second integral gives us the integral of columns 5 through 10.

Previous Posts In This Series:
CO2 Derivatives (Not Al Gore's Kind Of Derivatives)

CO2 Derivatives (Not Al Gore's Kind Of Derivatives)

A few days ago an article appeared on What's Up With That that claimed to have disproven AGW using statistics. That claim seems rather bold to me, but there were aspects of the paper I found interesting.

The paper talked about the "1st differences" and "2nd differences" of CO2. By this, the author meant the 1st and 2nd derivatives of CO2. Here I mean derivative as the term is used in Calculus, not the CO2 financial derivatives Al Gore wants to sell you on his carbon stock exchange.

This post shows how to calculate the 1st and 2nd derivatives of a variable in the Climate Scientist Starter Kit.

Calculating Derivatives In The Climate Scientist Starter Kit
It turns out it's pretty easy to calculate 1st and 2nd derivatives in the Climate Scientist Starter Kit. A derivative is calculated as dy/dx. In English, this means the change in y divided by the change in x. In this case, the term "y" refers to the y axis and "x" refers to the x axis. So you can calculate a derivative with the following spreadsheet formula:

=(Y_Cell - Previous_Y_Cell) / (X_Cell - Previous_X_Cell)

But its actually even easier than that. Because the X axis represents a series of months, the difference between any two adjoining X values is always 1. This simplifies the formula to:

=(Y_Cell - Previous_Y_Cell) / 1

And because the result of any number divided by one is that number, the formula simplifies even further:

=(Y_Cell - Previous_Y_Cell)

And there's our derivative!

Calculating the 2nd derivative is just the process of getting the derivative of the first derivative. In other words, it's just the same (Y_CELL - Previous_Y_Cell) run against the 1st derivative rather than the original data series.

The 1st And 2nd Derivative Of CO2
The graphs below show the CO2 data from the Climate Scientist Starter Kit, and the 1st and 2nd derivatives of that data.

CO2 Data

CO2 1st Derivative

CO2 2nd Derivative

The 1st derivative tells us the rate of change in the amount of CO2. The 2nd derivative tells us the rate of change in the rate of change in the amount of CO2.

Conclusion
Originally, I had planned to show how the 2nd derivative of CO2 has a good match with changes in cosmic rays. To do this you just lay the normalized cosmic rays on the graph with the 2nd derivative of CO2.

I had done this very quickly with a couple of decades of data when I first read the article I mentioned above. The match was very good. The 2nd derivative of CO2 and cosmic rays changed in lockstep with one another. Unfortunately, when I extended the analysis to the full range of data for the purposes of writing this post, the new range didn't have that nice correlation.

So I have no cool correlation to show you, but now you know how to calculate 1st and 2nd derivatives of data in the Climate Scientist Starter Kit. Well, ok, I do have one correlation to show you. It's between the 1st derivative of CO2 and the Solar Ephemeris. A similar correlation also exists with the 2nd derivative of CO2.

References:
New paper on mathematical analysis of GHG
Climate Scientist Starter Kit v1.5

NSIDC L2_Land Source Code Walkthrough

In this post we're going to take a closer look at the NSIDC source code that produces their Level 2B Land product. For a very brief overview of the Level 2B Land and other NSDIC source see this post.

Inputs
The source code uses the Level 2A Brightness Temperature file as well as several data files as input. The data files are stored in the anc folder that comes with the source.

NOTE: The data files in the anc folder are binary files, so if you want to see what data they contain, you'll need to write a translation program to convert them to ASCII text. See the the Fortran file ancdata.blk in the src/land/level2/common folder for a description of the data in each data file.

Source Code
The source code is located in the src/land/level2 folder. There are three main groups of source code: C code, Fortran code and common code located in the common folder. The C code is used mainly for QA and I/O purposes. It's the Fortran code that does the real work. The code in the common folder is also Fortran code, stored as reusable blocks.

NOTE: Recall from this post that the AMSR-E scans detects both the H and V polarization of light. The H and V values are what allows the source code to detect land, rain, wind, and other characteristics. It's the source code that actually detects these characteristics, not the satellite.

NOTE: You'll see the term TB used throughout the code. This is shorthand for Brightness Temperature.

The main file is amsre_l2main.f. This file will call all the other code as needed. Overall, the program performs the following processes:

1. Ingest ancillary databases and external parameters.
2. Ingest AMSR-E Level 2A TB data.
3. Grid TB data on EASE-grid projection.
4. Export gridded TB data.
5. Perform geophysical retrieval.
6. Export results/flags as Level 2B land product.

Other source files of note are:

• dobsonv3.f  Compute the dielectric constant of soil as a function of frequency, soil moisture, sand/clay fractions, and surface temperature.
• fmod_pr3ch.f Compute TB as a function of the 6.9, 10.7, and 18.7 frequencies.
• fmod_regrs.f Compute vegetation water content and soil moisture as a function of the 10.7, and 18.7 frequencies.
• fmod_tb4ch.f Compute R squared and Chi values based on the 10.7, and 18.7 frequencies and values calculated in dobsonv3.f.
• fmod_tb6ch.f Compute R squared and Chi values based on the 6.9, 10.7, and 18.7 frequencies and values calculated in dobsonv3.f.

Execution

AMSRE_L2MAIN() executes with 5 command-line arguments:

                  amsre_l2main.exe INPUT_L2A ANCIL_DIR L2B_DIR GTB_DIR PMC_VER

                INPUT_L2A: Filename of the input Level 2A data file

                ANCIL_DIR: Directory of ancillary databases

                L2B_DIR:   Directory of output Level 2B data files

                GTB_DIR:   Directory of gridded TB data files

                PMC_VER:   Product maturity code and PGE version number

NOTE: This program is designed to execute on big endian machines. If you're using a little endian machine (Intel machines are little endian) you'll need to add code that converts binary data to little endian values. You'll want to change the file l2ard.c and any code that loads ancillary data larger than than one byte. C++ code for swapping bytes is provided here:


inline bool is_bigendian()
{
const int i = 1;
bool bReturn = ( (*(char*)&i) == 0 );
return bReturn;
} // is_bigendian

inline char* reverse_16_bit (char* in16Bit)
{
char aReversed16Bit[2];

aReversed16Bit[0] = in16Bit[1];
aReversed16Bit[1] = in16Bit[0];
in16Bit[0] = aReversed16Bit[0];
in16Bit[1] = aReversed16Bit[1];

return in16Bit;
} // reverse_16_bit

inline char* reverse_16_bit_if_little_endian (char* in16Bit)
{
if (false == is_bigendian())
return reverse_16_bit(in16Bit);
return in16Bit;
} // reverse_16_bit_if_little_endian

inline char* reverse_32_bit(char* in32Bit)
{
char aReversed32Bit[4];

aReversed32Bit[0] = in32Bit[3];
aReversed32Bit[1] = in32Bit[2];
aReversed32Bit[2] = in32Bit[1];
aReversed32Bit[3] = in32Bit[0];
in32Bit[0] = aReversed32Bit[0];
in32Bit[1] = aReversed32Bit[1];
in32Bit[2] = aReversed32Bit[2];
in32Bit[3] = aReversed32Bit[3];

return in32Bit;
} // reverse_32_bit

inline char* reverse_32_bit_if_little_endian (char* in32Bit)
{
if (false == is_bigendian())
return reverse_32_bit(in32Bit);
return in32Bit;
} // reverse_32_bit_if_little_endian

inline char* reverse_64_bit(char* in64Bit)
{
char aReversed64Bit[8];

aReversed64Bit[0] = in64Bit[7];
aReversed64Bit[1] = in64Bit[6];
aReversed64Bit[2] = in64Bit[5];
aReversed64Bit[3] = in64Bit[4];
aReversed16Bit[4] = in64Bit[3];
aReversed64Bit[5] = in64Bit[2];
aReversed64Bit[6] = in64Bit[1];
aReversed64Bit[7] = in64Bit[0];
in64Bit[0] = aReversed64Bit[0];
in64Bit[1] = aReversed64Bit[1];
in64Bit[2] = aReversed64Bit[2];
in64Bit[3] = aReversed64Bit[3];
in64Bit[4] = aReversed64Bit[4];
in64Bit[5] = aReversed64Bit[5];
in64Bit[6] = aReversed64Bit[6];
in64Bit[7] = aReversed64Bit[7];

return in64Bit;
} // reverse_64_bit

inline char* reverse_64_bit_if_little_endian (char* in64Bit)
{
if (false == is_bigendian())
return reverse_32_bit(in64Bit);
return in64Bit;
} // reverse_64_bit_if_little_endian


template<class NUMBER_TYPE>
inline NUMBER_TYPE reverse_bytes(NUMBER_TYPE inValue)
{
if (2 == sizeof(inValue))
return *(reverse_16_bit((char*) &inValue));
if (4 == sizeof(inValue))
return *(reverse_32_bit((char*) &inValue));
if (8 == sizeof(inValue))
return *(reverse_64_bit((char*) &inValue));
return inValue;
} // reverse_bytes

template<class NUMBER_TYPE>
inline NUMBER_TYPE reverse_bytes_if_little_endian(NUMBER_TYPE inValue)
{
if (false == is_bigendian())
return reverse_bytes(inValue);
return inValue;
} // reverse_bytes_if_little_endian

Output

The Level 2B land product contains surface type, surface moisture, and vegetation water content information for the Earth.

Additional Information

Useful documentation on the algorithms is available at AMSR-E/Aqua L2B Surface Soil Moisture, Ancillary Parms, & QC EASE-Grids.

References:

First Look At NSIDC Source Code

Taking A Look At Raw AMSR-E Data

AMSR-E/Aqua L2B Surface Soil Moisture, Ancillary Parms, & QC EASE-Grids

Update On My Attempts To Get AIRS and UAH Source Code

UAH Source Code
I noted in this post that Dr. Christy of UAH had told me he was working with John Bates of NOAA to make UAH source code available. Following a suggestion by lucia from The Blackboard, I wrote to Mr. Bates asking him the status of the project.

Mr. Bates mistook me for a member of Dr. Christy's team, and therefore I don't think I should share his exact quotes with the world on a blog. I will say, however, that he confirmed the existence of the project and indicated that it is in its very early stages with no ETA at this time.

So it may be some time before we see UAH source code, but they are working on it.

AIRS Source Code
Recall from this previous post that AIRS tried to hide behind the International Traffic in Arms Regulations" (ITAR) as an excuse not to provide me their source code. My response to them was that I am a U.S. citizen with no plans to export the code, and therefore not covered by ITAR.

The AIRS Science Team Member I had been talking to forwarded the e-mail to someone else at NASA, apparently not on AIRS. Here's what they said.

the AIRS Product Generation Executable (PGE) software has not been cleared for public release for a variety of reasons.  It is available only to the members of the AIRS Science Team and those directly working for the AIRS Project.  Algorithms and validation results are published in the peer reviewed literature.  Data products and executable software are available as mentioned below.  Please see links below.

May I ask what your need is for the source code?  Perhaps there is some other way we can be of assistance?

So it seems I've gotten past the bogus ITAR claim and now they are hand waving with the excuse of "a variety of reasons" why the code couldn't possibly be made available to the public.

So I wrote them back, giving reason why only the source code can do what I need and asking them to specifically list what the "variety of reasons" are.

I intend to use the source code for personal education, mostly of items not covered in the literature, yet found in the data supplied by the AQUA team and other sources. 

As an example, the file AIRS.2010.01.01.240.L1B.AMSU_Rad.v5.0.0.0.G10005063822.hdf contains scan data on channel 5 that exceeds the limits of expected values defined in "The Limb Adjustment of AMSU-A Observations: Methodology and Validation". I'd like to know how the software handles such unexpected values and cannot learn that from the literature. The literature never expected such values and therefore does not cover how to handle them. Similarly, I cannot tell by running pre-compiled executables what happens when such unexpected values are encountered. Only the source code can tell me.

This is one example. There are others, such as experimenting with replacing the statistical error correction algorithms with algorithms that use deterministic data from other sources, such as cosmic ray monitors from Haleakala, Climax, Hermanus, South Africa, McMurdo, Inuvik, and other stations. I cannot get reliable results that can be meaningfully compared to existing products by attempting to reverse engineer the source code from the literature, nor can I use pre-compiled binaries for this purpose.

So I hope that you can help me in this area. As I told Mr. _______, I am a U.S. citizen and have no intention of exporting this code.

Can I ask you, sir, to list in detail what you mean when you say a "variety of reasons" exist that prevent a U.S. taxpayer from seeing computer code to process weather information that was created by funding from U.S. taxpayers? What exactly are these reasons?

Thank you for your time and help with this.

We'll see what they say.

Climate Scientist Starter Kit v2.0 Coming, Part III: Ozone And Pressure

In this post we look at two new data sets being added to version 2.0 of the Climate Scientist Starter Kit: Ozone and Tropospheric Pressure.

Ozone
Ozone data from the ISCCP project is being added in version 2.0. Because Ozone is an ISCCP product, it'll have regionalized versions of the data as well as a global version.


Tropospheric Air Pressure
Tropospheric Air Pressure data from the ISCCP project is being added in version 2.0. Because this is an ISCCP product, it'll have regionalized versions of the data as well as a global version.





Normalized Data Included
Normalized and reversed normalized data is included for both Ozone and Tropospheric Air Pressure. An example chart showing Normalized South Pole Air Pressure and reversed Normalized South Pole Ozone is shown below. Both datasets have a 13 month running mean added.

Previous Posts In This Series:
Climate Scientist Starter Kit v2.0 Coming
Climate Scientist Starter Kit v2.0 Coming, Part II: Regional Data

AMSR-E And AMSU HDF-EOS C++ Readers Are Done

The C++ code for reading AMSU Level 1B and AMSR-E Level 2A HDF-EOS files is complete. You can find the code here:

HDFReader.h
HDFReader.cpp
AMSUReader.h
AMSUReader.cpp
AMSREReader.h
AMSREReader.cpp

And this brings to an end the series of posts on being able to read UAH and AMSR-E raw satellite data.

Previous Posts In This Series:
Taking A Look At Raw AMSR-E Data
Taking A Look At Raw UAH Data
HDF Reader C++ Code Is Written!
Some Useful Climate Code
Summary Of Aqua Satellite Data, Computer Code, And Broken Equipment
Aqua Satellite Raw AMSR-E Data
Aqua Satellite Raw UAH Data, Part 2
Aqua Satellite Raw UAH Data, Part 1
Satellite Summary
Aqua Satellite Data Processing
A Note On The UAH And RSS Raw Data
How UAH And RSS Temperatures Are Measured
Overview Of The Aqua Satellite
Looking At The Aqua Satellite Data
UAH Satellite Data
Dangit! More Climate Stuff. UAH and RSS Raw Data

So, About That January UAH Anomaly

In a previous post I discussed that high anomaly reading from UAH's January measurements, pointing out that it's possible UAH's statistical adjustments to scan readings could have been thrown off by the unusual weather. In another post I discussed how the AMSU data used by UAH has known errors in nearly all of its scans. The error for channel 5, which is used to create the UAH anomaly readings, can go as high as -10 degrees K. In yet another post I discussed the raw scan line data that I've pulled from the AMSU on the Aqua satellite. It just so happens the data I pulled occurred during January.

So in this post, I want to combine those three posts and show why I'm still not convinced the January anomaly reading from UAH was correct. Basically, the data I pulled from January had an error of -14 degrees K in the channel 5 reading, which is 40% higher than the the maximum error according to the journals.

I can't help but wonder how well the statistical methods used to correct the known channel errors react to errors so far out of range of expected values. Actually, I can't help but wonder if errors of this size have ever been tested at all.

Which means I can't help but wonder if the January UAH anomaly has any validity at all.

It also makes me wonder how much additional data falls outside the expected error range. Because I didn't put any effort at all into finding data 40% beyond that range.

References:
Some Useful Climate Code
A Note On UAH's High January Temperature
Taking A Look At Raw UAH Data

First Look At NSIDC Source Code

In this post we'll take a look at how to order source code from the National Snow and Ice Data Center and what that source code does.

Ordering The Source Code
The source code is stored in 9 folders that NSIDC calls Delivered Algorithm Packages (DAPs). You must order the source code before you download it. This is done by sending off a request to their User Services department using this form. Just tell them you'd like to download the DAPs and they will give you instructions on how to register and download the DAPS via FTP.

Some of the DAPS also use the PORT Mathematical Subroutine Library.

The Code
The code you'll get is written in Fortran and C. There are also data files required by each DAP. Each DAP comes in its own folder and each folder contains a zip file of the source and data files, as well as some read-me files. After you unzip the files, you'll find the source code in the src folder and the data files in the anc folder.

To finish out this post, we'll take a quick look at each DAP.

L2_Land

Input File	Level 2A Brightness Temperature File
Output File	Level 2 Land File
Data Produced	Soil Moisture, Vegetation Water Content, Land Surface Temperature, Surface Type
Uses PORT?	Yes
Platform	Linux

L2_Rain

Input File	Level 2A Brightness Temperature File
Output File	Level 2 Rain File
Data Produced	Rain Rate, Rain Type, Rain Status, Surface Type
Uses PORT?	No
Platform	Linux

L2A

Input File	Level 1A AMSR-E File
Output File	Level 2A Brightness Temperature File
Data Produced	AMSR-E Scan Information
Uses PORT?	No
Platform	Windows 2000/XP

L2B_Ocean

Input File	Level 2A Brightness Temperature File
Output File	Level 2B Ocean File
Data Produced	Very Low Resolution Sea Surface Temperature, Low Resolution Sea Surface Temperature, Low Resolution Wind, Medium Resolution Wind, Medium Resolution Vapor, High Resolution Cloud
Uses PORT?	No
Platform	Linux

L3 Land

Input File	30 Days Of Level 2 Land Files
Output File	Level 2B Ocean File
Data Produced	AMSR-E Frequency Scans, Soil Moisture, Vegetation Water Content, Land Surface Temperature
Uses PORT?	No
Platform	Linux

L3 Ocean

Input File	Level 2B Land Files (Days Worth, Weeks Worth, Month's Worth)
Output File	Level 2B Ocean File
Data Produced	Low Resolution Sea Surface Temperature, Low Resolution Wind, Medium Resolution Wind, Medium Resolution Vapor, High Resolution Cloud
Uses PORT?	No
Platform	Linux

L3 Rain

Input File	Level 2A Brightness Temperature File (Month's Worth) Level 2B Rain Files (Month's Worth)
Output File	Level 3 Rain File
Data Produced	Land Rain, Ocean Rain
Uses PORT?	No
Platform	Linux

L3 Sea Ice

Input File	Level 2A Brightness Temperature File (Day's Worth)
Output File	Level 3 Sea Ice File
Data Produced	Frequency Scan Data, Ice Data, Snow Depth Data
Uses PORT?	No
Platform	Linux

L3 Snow

Input File	Level 2A Brightness Temperature File (Day's Worth), Level 3 Snow Files (5 Days Worth, Month's Worth)
Output File	Level 3 Snow File (Daily, 5 Days, Monthly)
Data Produced	Snow Data
Uses PORT?	No
Platform	Linux

References:
National Snow And Ice Data Center User Services Contact Form
PORT Mathematical Subroutine Library

Source Code!

Oh yeah! I have source code from the National Snow And Ice Data Center! URAH!

I've just downloaded it and haven't had the time to go over it yet (there's nine different groups of source). But details will follow.

Raw Data

Dr. John Christy On UAH Source Code

Following a suggestion by lucia over at The Blackboard, I wrote to Dr. Christy and Dr. Spencer of UAH asking about the public availability of the source code used to process UAH data. Dr. Christy replied:

We are in a program with NOAA to transfer the code to a certified system that will be mounted on a government site and where almost anyone should be able to run it.  We actually tried this several years ago, but our code was so complicated that the transfer was eventually given up after six months.

So UAH source code isn't currently available, but they're in the process of working with a NOAA program to make it available. I followed up asking if there was a general ETA for this availability.  He replied:

I talked with John Bates of NOAA two weeks ago and indicated I wanted to be early (I said the "first guinea pig") in the program.  He didn't have a firm date on when his IT/programming team would be ready to start the transition, so I don't know.

So that's where we stand. No source code now, but there is a process to make it available in the future.

I'd like to thank Dr. Christy for his help in this, and his quick replies (almost immediate!). He also provided me with the official boundaries used to create the various UAH regions in their temperature data, such as extra-tropic boundaries. I've updated the code that produces the ISCCP regional products to exactly match UAH boundaries. More information on that is here.

Previous Posts In This Series:
The Weather And International Traffic in Arms Regulations: Perfect Together

The Weather And International Traffic in Arms Regulations: Perfect Together

Behind the scenes I've been trying to get ahold of the source code for processing raw UAH weather data from the AMSR unit aboard the Aqua satellite. I sent off a letter to NASA asking for "the source code for producing the AIRS Level 2 standard retrieval product using AIRS IR and AMSU, without-HSB from Level 1 products. "

Here's the answer I got back:

The AIRS source code is ITAR controlled and is not cleared for public release.  All AIRS Products are available from the GES/DISC.  We also have direct broadcast software to allow local product generation.  More information is available at: http://airs.jpl.nasa.gov/data_products/data_products_toc/"

ITAR stands for "International Traffic in Arms Regulations". It covers exporting military weapons to foreign counties or to foreign individuals. The list of items it covers includes satellites and related equipment.  The link in their response, by the way, points to various tools, but no source code for actually processing temperature information.

However, ITAR is related to the export of these items. And as I'm a U.S. citizen with no plans to export the code, it's not clear to me how this covers my situation. I sent a response to NASA saying as much and asking for their explanation of the matter. We'll see what they have to say.

UPDATE:
Here are screen shots of the e-mails, with personal information blacked out.

Climate Scientist Starter Kit v2.0 Coming, Part II: Regional Data

Update:
The region boundaries described below have been updated to exactly match UAH region boundaries. My thanks to Dr. John Christy of UAH for supplying the boundary information.
======

A new feature in the Climate scientist Starter Kit v2.0 is regional data. For all ISCCP data products, the following regions will be supported:

• Global
• Northern Hemisphere (north of Equator)
• Southern Hemisphere (south of Equator)
• Tropics (70˚ from South Pole to 110˚ from South Pole)
• Northern Extra Tropics (110˚ from South Pole to 120˚ from South Pole)
• Southern Extra Tropics (70˚ from South Pole to 60˚ from South Pole)
• North Pole (120˚ from South Pole)
• South Pole (60˚ from South Pole)

This provides data similar to the UAH temperature data products and allows for analysis on the local level in addition the global level already supported. The ISCCP data in v1.5 includes Clouds and Water Vapor. In v2.0 we'll see the introduction of additional ISCCP data, which wil be discussed in a future post.

Splitting the ISCCP data into regions was done based on the girding of the data product. For equal area grids, the ISCCP data product covers the Earth using the method shown in the diagram below.

This grid was used to create the regional data products.

Cosmic rays will also have regional coverage. This will be done by adding the readings from additional monitors around the globe.

The graph below shows the v2.0 water vapor values as broken up by region.

Normalized Data Included

Normalized data is also included for the regional data. The graph below shows normalized North Pole Water Vapor along with normalized Solar Ephemeris.

Previous Posts In This Series

Climate Scientist Starter Kit v2.0 Coming

Taking A Look At Raw AMSR-E Data

In this post we're going to take a look at the data in the Level 2A AMSR-E file. But first, a short review on how the AMSR-E scans data.

The AMSRE-E scans at 6 frequencies: 6.9, 10.7, 18.7, 23.8, 36.5, and 89.0 GHz. There are 5 scanning resolutions. The first four scanning resolutions are considered "low res" and stored in an array of data that is 1994 by 243 positions. The fifth scanning resolution is "high res" and stored in an array that is 1994 by 486 positions.

There are also smoothed and unsmoothed versions of the data. The chart below shows which versions of data for each frequency are stored in the Level 2A file.

Link to original table

AMSR-E Spatial Characteristics of Observations

Reso-lution	Foot print	Mean spatial resolution	Channels
			89.0 GHz	36.5 GHz	23.8 GHz	18.7 GHz	10.7 GHz	6.9 GHz
1	75 km x 43 km	56 km	•	•	•	•	•	• o
2	51 km x 29 km	38 km	•	•	•	•	• o
3	27 km x 16 km	21 km	•	•	• o	o
4	14 km x 8 km	12 km	•	o
5	6 km x 4 km	5.4 km	o

•  Includes Level-2A (smoothed) data
o  Includes Level 1B (un smoothed) data at original spatial resolution

In addition to all this each scan has two components, a vertical (V) component and a horizontal (H) component. These different components are used to measure electric and magnetic values which in turn are used to make determinations about exactly what is being scanned (land, sea, ice, water vapor, etc.)

All of these variations are stored in the Level 2A AMSR-E file. The table below provides the variable names for all this data.

Level 2A Variable names

Variable Name	Data
6.9V_Res.1_TB_(not-resampled)	6.9 V Res 1
6.9H_Res.1_TB_(not-resampled)	6.9 H Res 1
10.7V_Res.2_TB_(not-resampled)	10.7 V Res 2
10.7H_Res.2_TB_(not-resampled)	10.7 H Res 2
18.7V_Res.3_TB_(not-resampled)	18.7 V Res 3
18.7H_Res.3_TB_(not-resampled)	18.7 H Res 3
23.8V_Approx._Res.3_TB_(not-resampled)	23.8 V Res 3
23.8H_Approx._Res.3_TB_(not-resampled)	23.8 H Res 3
36.5V_Res.4_TB_(not-resampled)	36.5 V Res 3
36.5H_Res.4_TB_(not-resampled)	36.5 H Res 3
6.9V_Res.1_TB	6.9 V Res 1 Smoothed
6.9H_Res.1_TB	6.9 H Res 1 Smoothed
10.7V_Res.1_TB	10.7 V Res 1 Smoothed
10.7H_Res.1_TB	10.7 H Res 1 Smoothed
10.7V_Res.2_TB	10.7 V Res 2 Smoothed
10.7H_Res.2_TB	10.7 H Res 2 Smoothed
18.7V_Res.1_TB	18.7 V Res 1 Smoothed
18.7H_Res.1_TB	18.7 H Res 1 Smoothed
18.7V_Res.2_TB	18.7 V Res 2 Smoothed
18.7H_Res.2_TB	18.7 H Res 2 Smoothed
23.8V_Res.1_TB	23.8 V Res 1 Smoothed
23.8V_Res.1_TB	23.8 H Res 1 Smoothed
23.8V_Res.2_TB	23.8 V Res 2 Smoothed
23.8H_Res.2_TB	23.8 H Res 2 Smoothed
23.8V_Res.3_TB	23.8 V Res 3 Smoothed
23.8H_Res.3_TB	23.8 H Res 3 Smoothed
36.5V_Res.1_TB	36.5 V Res 1 Smoothed
36.5H_Res.1_TB	36.5 H Res 1 Smoothed
36.5V_Res.2_TB	36.5 V Res 2 Smoothed
36.5H_Res.2_TB	36.5 H Res 2 Smoothed
36.5V_Res.3_TB	36.5 V Res 3 Smoothed
36.5H_Res.3_TB	36.5 H Res 3 Smoothed
89.0V_Res.1_TB	89.0 V Res 1 Smoothed
89.0H_Res.1_TB	89.0 H Res 1 Smoothed
89.0V_Res.2_TB	89.0 V Res 2 Smoothed
89.0H_Res.2_TB	89.0 H Res 2 Smoothed
89.0V_Res.3_TB	89.0 V Res 3 Smoothed
89.0H_Res.3_TB	89.0 H Res 3 Smoothed
89.0V_Res.4_TB	89.0 V Res 4 Smoothed
89.0H_Res.4_TB	89.0 H Res 4 Smoothed
89.0V_Res.5A_TB_(not-resampled)	89.0 V Res 5
89.0H_Res.5A_TB_(not-resampled)	89.0 H Res 5

The different frequencies are used to measure different attributes of the Earth, as shown in the table below.

Frequency Measurements

Frequency	Measurement
6.9	Sea Surface Temperature, Soil Moisture, Vegetation
10.7	Sea Surface Temperature, Wind Speed
18.7	Wind Speed, Water Vapor
23.8	Unknown
36.5	Cloud Liquid Water
89.0	Rain Rate

With the above information, you can pull the desired data from the Level 2A file. The data for an actual file for 36.5V_Res.4_TB_(not-resampled), which the cloud liquid water frequency,  is shown below. It shows a single scan line.

References And Previous Posts In this Series:
AMSRE_4_NWS.ppt
Taking A Look At Raw UAH Data
HDF Reader C++ Code Is Written!
Some Useful Climate Code
Summary Of Aqua Satellite Data, Computer Code, And Broken Equipment
Aqua Satellite Raw AMSR-E Data
Aqua Satellite Raw UAH Data, Part 2
Aqua Satellite Raw UAH Data, Part 1
Satellite Summary
Aqua Satellite Data Processing
A Note On The UAH And RSS Raw Data
How UAH And RSS Temperatures Are Measured
Overview Of The Aqua Satellite
Looking At The Aqua Satellite Data
UAH Satellite Data
Dangit! More Climate Stuff. UAH and RSS Raw Data