magicjava

Winding Down

2010-12-10T09:07:00.000-05:00

Just a short post to announce I'll be wrapping up this blog in the near future. This is mostly due to the complete collapse of the environmental movement's credibility. While the movement still exists and there are still battles to be fought, there are plenty of people capable of doing it without my help. This allows me to turn my attention to matters in my personal world.

I hope to get the Climate Scientist version 2.0 out and check NASA numbers from the FOIA stuff, and wrap up any other loose ends that come to mind. But these updates will come only as time permits and once they're done, so too will this blog be done.

NASA Responds To FOIA Request

2010-12-10T03:05:00.001-05:00

NASA has responded to my FOIA Request. To recap, I requested two items from NASA: 1) the height in the atmosphere scanned by each footprint for each scan of the Aqua satellite, and 2) the Ai and Theta Bar i used by NASA to calculate the synthetic values for Aqua's channel 4.

Their response was that there are no records for the height of the footprint scans on the Aqua satellite and that the values for Ai and Theta Bar i are as follows:

A(i) = 0.38648412E-1, -0.13865858, 0.60949275, 0., 2.0037199, -0.95993573, 0., 0.28063577, -0.9307997E-1, 0.37032401E-1, -0.15795620E-2, -0.49943969E-2, -0.11815043E-1, 0.70557197E-1, -0.33432437E-1

thetabar(i) = A_mean(i) * amsu_noise_tr(i) where:

A_mean(i) = 1200.2806, 901.08093, 949.21509, 0., 1242.5841, 1382.2343, 0., 1359.1735, 1323.4860, 975.67328, 920.14246, 654.80048, 499.30478, 313.854318, 1955.0244

amsu_noise_tr(i) = 0.17, 0.22, 0.25, 0.14, 0.20, 0.17, 0.14, 0.16, 0.16, 0.22, 0.24, 0.35, 0.48, 0.80, 0.12

Notice that each of these collections of numbers contains 15 numbers. These correspond to the 15 channels on the Aqua satellite.

I still need to check that the above values, when used in NASA's formula for creating synthetic channel 4 data, actually produce the published values for that channel.

I'm out of ideas on how to obtain the height in the atmosphere each footprint scans.

Previous Posts In This Series
Appealing NASA's FOIA Decision
NASA Responds To FOIA Request
Quick FOIA Update
The Government Way
FOIA Request Filed With NASA

Appealing NASA's FOIA Decision

2010-05-04T00:59:00.003-04:00

I've decided to appeal the FOIA decision from NASA, JPL. To do this I have to write out a hard copy letter and snail mail it to the appeals office. Here's the copy of that letter:

Dear Sirs,

I'm writing this letter to appeal a FOIA decision involving NASA JPL. The decision involved 3 issues, the first of which has been resolved. For the second and third decision, I've included relevant background information as well as the reason I'm appealing.

Issue 1: Radiative Transfer Algorithm Used By AIRS.
Status: Resolved.

Issue 2: Atmospheric Scan Depth Of Aqua Satellite AMSU Channel 5 Footprints.

Background
The Aqua satellite has an AMSU instrument that scans the atmosphere at 15 different channels. Each of these channels scans 30 different locations in the atmosphere. These locations are sometimes referred to as "footprints". Each footprint scans at a different height in the atmosphere.

NASA Discussion On How This Data Is Used
Where reliable sensor data is available, it is applied directly to the appropriate portion of the atmosphere, taking into account the angle of the observation.
- Steven Friedman, NASA JPL, Personal Correspondance (Included)

My FOIA Request
(2) Atmospheric scan depth for each footprint on channel 5 of the AQUA AMSU.

FOIA Response
This is to advise that NASA has no responsive Government records at JPL for parts (2) and (3) of your request.

Why I'm Appealing This Decision
NASA JPL has indicated that they apply each scan to the appropriate portion of the atmosphere. This is impossible to do if it is unknown at what depth of the atmosphere a given footprint is scanning.

Issue 3: The 230000 Scan Readings And/Or The Values Of Vectors Ai And Theta Bar i Used To Synthesis Aqua AMSU Channel 4 Data

Background
Aqua's AMSU channel 4 failed in late 2007, as described in AMSU-A Channel 4 NeDT Update: 20 December 2007 at the document archive. The data for channel 4 is now artificially generated. The generation process uses two vectors, Ai and Theta Bar i. The values for these vectors is not available to the public. The data for the vectors was itself generated from 230000 scans taken from the AMSU. This data is also not available to the public.

NASA Discussion On How This Data Is Used
See AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4 at http://disc.sci.gsfc.nasa.gov/AIRS/documentation/v5_docs/AIRS_V5_Release_User_Docs/V5-Modification-for-AMSU-Ch-4-NeDT.pdf for a detailed discussion on this data. In particular, see equation (1) for the use of Ai and Theta Bar i.

My FOIA Request
(3) In references to the creation of synthetic readings for the AQUA AMSU channel 4, the 230000 cases used to create the values for the vectors Ai and Theta Bar i, or the values of vectors Ai and Theta Bar i themselves if the 230000 readings are no longer available. These values and readings are referenced but not actually provided in the document AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4 available online at http://disc.sci.gsfc.nasa.gov/AIRS/documentation/v5_docs/AIRS_V5_Release_User_Docs/V5-Modification-for-AMSU-Ch-4-NeDT.pdf..."

FOIA Response
This is to advise that NASA has no responsive Government records at JPL for parts (2) and (3) of your request.

Why I'm Appealing This Decision
NASA JPL has indicated that this data is used to generate the synthesized channel 4 data. This would be impossible to do if the values for vectors Ai and Theta Bar i were not known to them, as these values are two of the four values used to generate the synthesized data.

Note that NASA JPL also indicated that NASA GSFC may have responsive records for issue 3. I am in contact with them to see if this is the case, but as my ability to appeal this decision is limited to 30 days, I am simultaneously appealing.

Thank you for you time and help in this matter.

...and the e-mail sent to NASA, GSFC for the data used to generate synthetic channel 4 values:

I was informed by Dennis B. Mahon of NASA, JPL that NASA GSFC may have information regarding data used to generate synthetic channel 4 data for Aqua's AMSU channel 4. Specifically, I am looking for he 230000 cases used to create the values for the vectors Ai and Theta Bar i, or the values of vectors Ai and Theta Bar i themselves if the 230000 readings are no longer available. These values and readings are referenced but not actually provided in the document AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4 available online at http://disc.sci.gsfc.nasa.gov/AIRS/documentation/v5_docs/AIRS_V5_Release_User_Docs/V5-Modification-for-AMSU-Ch-4-NeDT.pdf..."

See equation (1) of the referenced PDF for specific details.

Please note that this request is associated with a FOIA involving NASA, JPL that is currently under appeal.

Previous Posts In This Series
NASA Responds To FOIA Request
Quick FOIA Update
The Government Way
FOIA Request Filed With NASA

Aqua Satellite Project, Update 10 Released.

2010-05-02T22:32:00.003-04:00

Update 10 for the Aqua Satellite Project is ready. You can download it here. This update is part 2 of a 2 part release that adds global griding of the data.

This release adds the ISCCPSummarize program that summarizes ISCCP grid-style data generated by AMSUToGrid and the ISCCPNormalize program that normalizes output files from ISCCPSummarize on a scale of 0.0 to 1.0.

This release also fixed a bug in the AMSUToGrid class DegreeGrid that was passing grid edge rather than grid center to weights, replaced custom code in DegreeGrid to calculate grid locations with code that uses official ISCCP values, and added mask images and generated grid data to MapToGrid folder. Masks are in the Masks subfolder, generated grid data is in the Generated Grid Data subfolder. There are Masks and generated data for Africa, Antarctica, Arctic Ocean, Asia, Atlantic Ocean, Australia, Europe, Indian Ocean, Land and Water, North America, Pacific Ocean, South America, Southern Ocean, and US Lower 48 States. Generated data is in text, XML, JavaScript, and C code formats.

Regional Data
In addition to summarizing data for the entire globe, the ISCCPSummarize program provides summaries for the following regions:

Africa

Antarctica

Asia

Australia

Europe

North America

South America

Arctic Ocean

Atlantic Ocean

Indian Ocean

Pacific Ocean

Southern Ocean

...and the following UAH-like regions:

● Global Land
● Global Ocean
● Northern Hemisphere
● Northern Hemisphere Land
● Northern Hemisphere Ocean
● Southern Hemisphere
● Southern Hemisphere Land
● Southern Hemisphere Ocean
● Northern Extra Tropics
● Northern Extra Tropics Land
● Northern Extra Tropics Ocean
● Southern Extra Tropics
● Southern Extra Tropics Land
● Southern Extra Tropics Ocean
● North Pole
● North Pole Land
● North Pole Ocean
● South Pole
● South Pole Land
● South Pole Ocean
● U.S.A. Lower 48 States

These UAH regions use the same latitude boundaries as genuine UAH regions do. My thanks to Dr. Christy of UAH for providing me with the UAH region boundary information.

Three-Valued Logic And Irreproducible Results In Science, Part II

2010-04-29T06:29:00.022-04:00

Introduction
In the previous post for this series we discussed how Two-Valued and Three-Valued Logic works. In this post we take a look at how Three-Valued Logic (3VL) interacts with the results of science. We'll be using the properties of 3VL's three truth values, True, False, and Unknown, to do this.

In particular we'll be examining how the truth value Unknown of 3VL spreads through science like a virus. In fact, we'll make the claim that scientific results that have unverifiable claims that cannot be independently reproduced are viruses. Any further work that builds upon them is infected and becomes unverifiable and irreproducible itself.

Scientific Claims Must Be Verified

In science, it's not enough to simply make a claim. Claims must be independently verifiable before they are accepted.

When the procedures of the claim are followed and the results of the claim are reproduced, the claim is said to be True. When the procedures of the claim are followed and the results of the claim are not reproduced, the claim is said to be False. An example of a scientific claim being shown False is the original Cold Fusion hypothesis by Martin Fleischmann and Stanley Pons.

However, in situations where the procedures of the claim cannot be followed, we cannot show the claim to be True or False. It is in these cases that the need for 3VL arises, and the truth of the claim is Unknown.

Technical Aside: Cold fusion research has continued since the days of Fleischmann and Pons. Today it is funded by several governments around the world and many researchers in the field believe that the original claims of Fleischmann and Pons have been vindicated. However, several mainstream scientific organizations still disagree, most notably the U.S. Department of Energy and Science magazine. For more information on the ongoing cold fusion research, see the LENR-CANR website.

Example: Aqua Satellite Channel 4 Virus
It helps to have an example, so we'll be using channel 4 of the AMSU on the Aqua satellite. Channel 4 failed completely around December, 2007. In response to this, NASA created a new algorithm and has used it to synthetically create channel 4 data from October 1st, 2007 onward.

While NASA publishes the algorithm used to create synthetic channel 4 values, that algorithm requires certain data that is not available to anyone outside of NASA. Even the folks at NASA's JPL, who are in charge of the Aqua satellite, have said they don't have access to the data.

Without this data it's impossible to verify if the algorithm for synthesizing channel 4 data is correct, even though the algorithm itself is published. Similarly, we cannot demonstrate that the algorithm fails to correctly synthesis channel 4 data. Therefore, the ability of the algorithm to correctly synthesis data must be classified as Unknown because the statement that the algorithm is accurate cannot independently be shown to be True or False.

How The Virus Spreads
To qualify as a virus, the Unknown values must be capable of spreading to other works. To see how this occurs, let's first take a look at how False research is capable of spreading.

We take the example of research attempting to build upon claims that have been demonstrated False, in this case Cold Fusion. The diagram above shows new research that is correct being combined with the results of Cold Fusion. Because Cold Fusion has been shown to be False, the overall conclusions of the research must be False because they require Cold Fusion.

Logically, this situation is captured by a simple predicate: True AND False = False.

The same situation occurs in 3VL when using Unknown, rather than False, values. Because the creation of synthetic data for Aqua's AMSU Channel 4 cannot be shown to be True or False, it is Unknown. Any research combined with it, no matter how good it is, produces a final result that is also Unknown.

This too is captured by a simple 3VL predicate: True AND Unknown = Unknown.

A concrete example of the spreading of Unknown results in published research is provided by NASA's claims of increased yield due to synthetic channel 4 data. We'll assume that these claims are True and that yields are in fact increasing. However, even with this assumption, we cannot demonstrate that yields should be increasing. Because it cannot be verified that the synthetic channel 4 data is valid, we cannot verify that the synthetic data causes bad data to pass QA or good data to fail QA. The quality of the data in these increased yields is Unknown. This is because the quality of the synthetic data is Unknown.

This cascading of the Unknown value continues through anything that uses the data from these increased yields. In practice, it turns out that all processes referred to by NASA as "Level 2" or higher that use Aqua AMSU data will be infected by the Unknown values. That is, all such data sets have an Unknown truth value themselves due to their dependence upon the increased yield data. These "Level 2" products include:

● Temperature profile from 3 mbar (45 km) to the surface.
● Water vapor profiles.
● Snow and ice coverage.
● Cloud liquid water.
● Cloud-cleared IR radiances.
● Rain Rate.
● Ozone.
● Carbon Dioxide Support Products.

Other Examples Of The Virus Spreading
GHCN Data

A while back Willis Eschenbach made the claim that GHCN data at Darwin station was being manipulated to show a warming where none existed in the raw data. Pro-AGW bloggers jumped on this claiming the adjustments were valid.

The problem is the Australian CSIRO Atmospheric Research Center provided no reason why the adjustments were made, stating only that Darwin is a urban site (which should make adjustments go down, not up).

This is an illustrative example of the problem with GHCN adjustments. Even though GHCN provides its raw data and describes its adjustment procedures, it's adjustments cannot be replicated when reasons for the adjustments aren't given.

For this reason, the validity of GHCN data must be classified as Unknown. This Unknown value spreads to anything using GHCN data. This includes the Intergovernmental Panel on Climate Change 4th Assessment Report.

Deep Impact

Deep Impact was a NASA mission to probe a comet by slamming a probe into the comet Tempel I and analyzing the impact. The Deep Impact team at NASA JPL released a photo of the comet with water photoshopped onto its surface (seen at left), a series of medium resolution images of the event, and a chart of the thermal emission spectra of the debris. The chart is shown below.

However, the chart is made of data that's been modified by NASA and the raw data used to generate the Tempel I spectra has never been released. This makes it impossible to verify that the scanners actually produced the results shown in the chart. The resulting unverifiable claims are therefore Unknown.

Curing The Virus
As far-reaching as the consequences of the examples provided here are, we've covered only a small handful of examples. Many more could be provided. Their flow through related work could be tracked and we'd discover that a significant portion of modern science rests upon unverifiable claims.

I think most people wouldn't consider such science to be science at all, but as a problem that stands in the way of science. Fortunately, it's a problem that's easy to fix.

Simply make the claims verifiable.

By making the raw data and computer code used to generate the claims publicly available, and by noting why changes are made to raw data, claims that are currently unverifiable can be demonstrated to be True or False.

And that is the whole reason science exists in the first place.

References
Three-Valued Logic And Irreproducible Results In Science, Part I
Cold Fusion claims by Martin Fleischmann and Stanley Pons - Wikipedia Entry
NASA Responds To FOIA Request
AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4
AMSU - Wikipedia Entry
AIRS/Aqua Level 2 Carbon Dioxide Support Products
The Smoking Gun At Darwin Zero
Willis Eschenbach caught lying about temperature trends
Updating Australia’s high-quality annual temperature dataset
GHCN V.2 Raw Data
GHCN Quality Control, Homogeneity Testing, and Adjustment Procedures
GHCN-Monthly Version 2 Introduction
Deep Impact
LENR-CANR website

Aqua Satellite Project, Update 9 Released.

2010-04-25T22:21:00.003-04:00

Update:
According to stats at SourceForge, this is the most popular release of the Aqua Satellite Project yet. To the folks downloading the software: feel free to tell me of any bugs you may find or improvements you'd like to see.
============

Update 9 for the Aqua Satellite Project is ready. You can download it here. This update is part 1 of a 2 part release that adds global griding of the data. This release adds the AMSUToGrid program for converting extracted csv files to a globally gridded format. The csv files must be extracted with longitude and latitude data. The output format is identical to the format used by the ISCCP project for 2.5 degree equal area grid data in ASCII format. The following weighting options are available as command line switches:

● DropInTheBucket
● InverseDistance
● LinearInterpolation
● SinXOverX
● CubicConvolution

Three-Valued Logic And Irreproducible Results In Science, Part I

2010-04-24T07:11:00.005-04:00

Introduction
This is a short series of two posts that discusses Three-Valued Logic (3VL) and uses it to demonstrate how irreproducible results corrupt not only the scientific work in which they appear, but also spread that corruption to any related work.

In short, this series of posts demonstrates that irreproducible results are viral. My current work on the Aqua satellite is used as an illustrative example in the second of these posts.

In this post we look at Two-Valued Logic and Three-Valued Logic and briefly discuss how they work.

Two-Valued Logic
To start off, let's take a quick look at the more common two valued logic system. This system gets its name from the fact that there are only two possible values for any statement: True or False. Two-Valued Logic is also called Boolean logic.

In regards to this post, we're most concerned with how these logic values of True and False can move through a system of logical predicates. For example, if we have a True statement and a False statement and a logical predicate requiring at least one true statement, than that predicate transforms the one True statement and one False statement into a single statement that is True. This is because the requirement of at least one True statement has been met and the predicate is therefore True.

Such a predicate is commonly referred to as an OR statement. For an OR statement to be True, either statement 1 or statement 2 must be True. There are several types of predicates in Boolean logic. The most common are OR, AND, and NOT. One way to define predicates is using a true table. A truth table shows all possible inputs to the predicate and what the output of the predicate is. The truth tables for OR, AND, and NOT in Boolean logic are shown below.

OR Truth Table.

AND Truth Table

NOT Truth Table.

In a nutshell, the OR predicate accepts two inputs and converts them to a single output. That output will be True if either of its inputs are True, or False if both inputs are False.

The AND predicate also accepts two inputs, but returns TRUE only if both inputs are True. In all other cases it returns False.

The NOT predicate simply flips True inputs to False outputs and False inputs to True Outputs.

TECHNICAL ASIDE: Truth tables are one way to define logical predicates. They use "case analysis", a method of listing every possible case and the corresponding outcome. However, there are other, more elegant ways to define predicates. In his book Predicate Calculus And Programming Semantics, the late Edsger Dijkstra defined the AND predicate using what is known as The Golden Rule, shown below, with the equivalence operator (==) having the lowest binding and meaning "Is The Same As":

The Golden Rule: p AND q == p == q == p OR q

This type of definition is considered superior to case analysis. Once the properties of the various predicates are captured in this manner, they can be used as building blocks to far more sophisticated theorems. Case analysis, on the other hand, can never demonstrate anything more than what is provided by the cases themselves.

Three-Valued Logic
Three-Valued Logic (3VL) builds upon Boolean Logic. There are several conventions for 3VL. We'll be looking at a particular convention, one in which the third truth value is Unknown. So 3VL, as discussed here, has three truth values: True, False, and Unknown.

TECHNICAL ASIDE: Relational database products make heavy use of 3VL, and refer to the Unknown truth value as NULL. The late E.F. Codd invented relational databases while working for IBM, He introduced the convention of using the word NULL to represent Unknown, and assigned NULL the symbol of the Greek lowercase omega, shown in the image at the beginning of this post.

Like Boolean Logic, 3VL has predicates that can be defined using truth tables. The truth tables for 3VL AND, OR, and NOT predicates are shown below.

When the only values in use are True and False, 3VL gives the exact same answers as Boolean logic. However, when an Unknown truth value is used as inputs it can also show up as the outputs for a predicate. The result of the predicate "False OR Unknown" is Unknown. The result of the predicate "NOT Unknown" is always Unknown. The result of an AND predicate with an Unknown input is always Unknown or False, never True.

The Point Of All This
It is this last result that interests us here. The result of the predicate "X AND Unknown" can never be True, no matter what the value of X is.

This is our stopping point for now. The next post in this series will discuss the relationship between Three-Valued Logic and science.

References
Boolean Logic - Wikipedia Entry
Ternary Logic - Wikipedia Entry
SQL NULL - Wikipedia Entry
Predicate Calculus And Programming Semantics - Edsger Dijkstra
A Relational Model of Data for Large Shared Data Banks - Edgar F. Codd

SEC Watched Porn As Economy Crashed

2010-04-22T23:35:00.000-04:00

I try to stay clear of politics on this blog, but every once in a while something happens that I just can't let slip by. This is one of those times.

The AP is reporting that the SEC's inspector general is conducting probes into 31 of its employee's charged with downloading and viewing porn on government time, using government equipment, while the economy crashed.

Here are a few highlights from the report:

● A senior attorney at the SEC's Washington headquarters spent up to eight hours a day looking at and downloading pornography. When he ran out of hard drive space, he burned the files to CDs or DVDs, which he kept in boxes around his office. He agreed to resign, an earlier watchdog report said.

● An accountant was blocked more than 16,000 times in a month from visiting websites classified as "Sex" or "Pornography." Yet he still managed to amass a collection of "very graphic" material on his hard drive by using Google images to bypass the SEC's internal filter, according to an earlier report from the inspector general.

● Seventeen of the employees were "at a senior level," earning salaries of up to $222,418.

For more details, see the original story here.

Aqua Satellite Project, Update 8 Released. Now With More She-Hulk!

2010-04-18T21:40:00.004-04:00

Update 8 for the Aqua Satellite Project is ready. You can download it here. This update is part 2 of a 2 part release that adds some spit and polish to the graphics, charting, and platform-independant font engines. Highlights for this release include:

● Added event handling to all shapes classes. These classes can now respond to mouse, keyboard, touch, and command events.
● Added anti-aliasing capabilities to shapes and FontManager. Anti-aliasing is turned on by default. It can be turned off by setting a flag in the various graphics and text calls.
● Added font scaling capabilities to the FontManager. The built-in fonts can now be used at any desired font size.
● Added histogram, bi-linear interpolation, equalize, threshold, grayscale, edge detection, translation, rotation, scaling, and erosion filters for use with bitmaps. These filters are located in the new Filters.h and Filters.cpp files in the Graphics folder.
● Made significant enhancements to the DrawingContext2D and Color classes.
● Fixed several bugs in the Chart class.

The code has been placed in the public domain.

Samples

Edge Detection

Equalize

Grayscale

Threshold

Rotate

Translate

Scale

Scaled, Anti-Aliased, Platform-Independant Text

Erosion

Bi-Linear Interpolation

Original Text Image (For Comparison)

Anti-Aliasing

Area Chart

Bar Chart

Line Chart

Pie Chart

Point Chart

NASA Responds To FOIA Request

2010-04-16T08:06:00.002-04:00

NASA has responded to my FOIA request.

In a nutshell, they provided me with a link to the AMSU-A Radiative Transfer Algorithm documentation and said they had no information on the scan depths for the footprints of channel 5 on the AMSU and didn't have the vector data they use to synthesis AMSU channel 4.

Both of the claims of "no information" seem impossible to me, as they're needed to process AMSU data the way NASA JPL does.

While I mull over my next move, here's their response.

Your Freedom of Information Act (FOIA) request for release of information from the files of the National Aeronautics and Space Administration (NASA) received in FOIA processing at the NASA Management Office-Jet Propulsion Laboratory on April 8, 2010. You requested the following:

"...(1) Documentation on how the AQUA AMSU-A Radiative Transfer Algorithm works. (2) Atmospheric scan depth for each footprint on channel 5 of the AQUA AMSU. (3) In references to the creation of synthetic readings for the AQUA AMSU channel 4, the 230000 cases used to create the values for the vectors Ai and Theta Bar i, or the values of vectors Ai and Theta Bar i themselves if the 230000 readings are no longer available. These values and readings are referenced but not actually provided in the document AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4 available online at http://disc.sci.gsfc.nasa.gov/AIRS/documentation/v5_docs/AIRS_V5_Release_User_Docs/V5-Modification-for-AMSU-Ch-4-NeDT.pdf..."

This is to advise you that responsive records to part (1) of your request may be found at the following web site:

http://eospso.gsfc.nasa.gov/eos_homepage/for_scientists/atbd/docs/AIRS/atbd-airs-L2.pdf

This is to advise that NASA has no responsive Government records at JPL for parts (2) and (3) of your request. Additionally, pertaining to part (3) of your request, we have no responsive records because the specific research and development was performed by the NASA Goddard Space Flight Center (GSFC). Therefore, it is possible that GSFC may have responsive records.

You have appeal rights concerning these actions.

And I do want to thank them for not charging me for this request. :)

Previous Posts In This Series
Quick FOIA Update
The Government Way
FOIA Request Filed With NASA

Aqua Satellite Project, Update 7 Released

2010-04-11T17:14:00.001-04:00

Update 7 for the Aqua Satellite Project is ready. You can download it here. This update is part 1 of a 2 part release that adds some spit and polish to the graphics, charting, and platform-independant font engines.

This release adds all the functionality that wasn't yet written for the Chart class and removes the need for external files from the font engine (although you can still use such files to add your own fonts, they are no longer required for the fonts shipped with the font engine). It adds a FontManger that manages the built-in fonts, using less memory, less disk space, and runs faster than the old method. As before, the built-in fonts are Arial, Courier, Helvetica, Times New Roman, TrebuchetMS, and Verdana in sizes of 10, 12, 18, 24, 36, 48, and 72. A bug in the Courier font bitmaps was also fixed.

The LiveGraph application no longer has a -font parameter as it can now use the FontManager to produce the fonts it needs.

All code has been placed in the public domain.

Draw2D V1.1 JavaScript Graphics Engine Update Released

2010-04-08T16:59:00.001-04:00

Version 1.1 of the Draw2D JavaScript Graphics Engine has been released. This is a minor update that adds speed improvements to all the curve drawing classes (Quadradic Curves, Cubic Curves, Round Rectangles, Ovals And Arcs) and fixes a bug in the rendering of Cubic Curves.

You can get the source code, several example HTML pages, and a Programmer's Manual here.

The code has been placed in the public domain.

Previous Posts In This Series
Draw2D JavaScript Graphics Engine Released

Quick FOIA Update

2010-04-08T15:46:00.001-04:00

Just dropping a note to update the status of my FOIA. In the previous post I noted that NASA JPL wanted me to state up front how much I was willing to pay for the information.

Having no idea what these fees are for, I figured the next step is to ask why they charge these fees. That way I can give some sort of reasonable answer on what I'd pay. Here's my resonse:

Dear Mr. Mahon,

As I mentioned in my previous e-mail, this is my first FOIA request, So, if I may, I'd like to ask the purpose of the fees. Are they to cover mailing costs? Will I be actually paying the salary of the person looking up the information? Are they arbitrary fees designed to prevent "fishing expeditions? The reason I ask is so that I can provide you with an answer that is sensible for the purpose of the fees. If possible, providing an "average fee" for obtaining one document would be useful.

Previous Posts In This Series
The Government Way
FOIA Request Filed With NASA

The Government Way

2010-04-07T23:15:00.000-04:00

I got a response back from NASA JPL regarding my FOIA request. They want to know up front just how much I'm willing to pay for the information. Gotta love how our government operates. :)

Here's the text (emphasis mine):

Greetings:

Please respond with your complete postal mailing address, phone and fax numbers. Also, you must state your willingness to pay fees that may be assessed in processing your request. Please reply stating your willingness to pay fees and the amount you are willing to pay so we may proceed with your request. Thank you.

So now I'll have to mull over just how much I'm willing to pay for information my taxes already paid for.

FOIA Request Filed With NASA

2010-04-06T23:49:00.000-04:00

About three weeks ago, as I noted in this post, I sent off a request to NASA JPL for information needed to reproduce their synthetic creation of AMSU channel 4 data and for the atmospheric scan depths of each footprint for channel 5 on the AMSU.

I've not received a response back, and 3 weeks is one week longer than my usual waiting period of two weeks. So today I filed a FOIA with NASA JPL for the information. A copy of the request is shown below.

Dear Mr. Mahon,

This is my first FOIA request, so allow me to apologize in advance for any procedural mistakes I may make.

About three weeks ago (on or about March, 13th, 2010) I requested several pieces of information from NASA JPL via the "Ask AIRS" web interface and have received no response. So I am now requesting that information via the FOIA. The requested information is:

● Documentation on how the AQUA AMSU-A Radiative Transfer Algorithm works.

● Atmospheric scan depth for each footprint on channel 5 of the AQUA AMSU.

● In references to the creation of synthetic readings for the AQUA AMSU channel 4, the 230000 cases used to create the values for the vectors Ai and Theta Bar i, or the values of vectors Ai and Theta Bar i themselves if the 230000 readings are no longer available. These values and readings are referenced but not actually provided in the document AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4 available online at
http://disc.sci.gsfc.nasa.gov/AIRS/documentation/v5_docs/AIRS_V5_Release_User_Docs/V5-Modification-for-AMSU-Ch-4-NeDT.pdf

Thank you for your time.

P.S.

Just in case you're wondering where my normal two-week waiting period came from, two weeks is the amount of time it took Stephen Hawking to respond to a question I had regarding black holes and quantum physics. Given Dr. Hawking's well-known medical issues, this should be a reasonable time frame for any reply from anyone else.

Aqua Satellite Project, Update 6 Released

2010-04-04T21:26:00.007-04:00

Update 6 for the Aqua Satellite Project is ready. You can download it here. The big item is the new LiveGraph program which creates a movie of a graph. This movie can be overlaid on top of GridToMap movies.

A sample LiveGraph movie is shown below. It graphs cloud and surface temperature data over a map of clouds.

Movie made from LiveGraph and GridToMap.

In support to LiveGraph, a graphics engine, a charting engine, and a platform-independant font engine have been added to the code. There was also some reworking of the MathUtils.h file to make it a lot smaller. Most of the code was moved to a new Math folder.

All code has been placed in the public domain.

What Ozone Hole?

2010-03-29T13:33:00.005-04:00

As a result of now being able to make movies from gridded data, I've been going through the ISCCP data that comes with the Climate Scientist Starter Kit. A movie of the ozone data is provided above.

What surprised me was the amount of ozone over Antarctica was equal to or greater than the amount of ozone over nearly the entire rest of the world. Only the Arctic consistently had more ozone (and it has quite a bit of ozone). This was true from the beginning of the movie (July, 1983) to the end (June, 2008).

Which makes me wonder why we've never heard of of an ozone hole over Africa, or South America, or India, all of which always have less ozone than Antarctica.

Now, I'm hardly an ozone expert. So if anyone out there would like to enlighten me on why this all makes sense, please feel free to do so. Because right now I'm wondering if the Ozone Hole is as big a hoax as Global Warming. Tips on why the Arctic has so much ozone are also welcome.

P.S.
If you watch the video all the way through, you get to see what happens when a satellite sensor goes batty.

Aqua Satellite Project, Update 5 Released

2010-03-28T18:41:00.007-04:00

Movie made from GridToMap images.

Update 5 for the Aqua Satellite Project is ready. You can download it here. The big item is the new GridToMap program which lets you create bitmap images from gridded data.

GridToMap takes various pieces of input data and creates an output bitmap. The inputs are:

• Climate data for the entire world in 2.5 by 2.5 degree gridded format. This is a .cvs file with 1 optional header row and 3 required header columns.
• A text pallet file describing which colors get assigned to which ranges of data.
• A legend bit map that can display whatever information you want, but is intended to show the image's legend. The legend bitmap must be 20 pixels high and the same width as the main map image.
• An optional superimpose bitmap that will be laid over the map. The superimpose image must be the same width and height as the map (not counting the 20 pixels for the legend).
• The map can begin at the Prime Meridian or the International Date Line.
• Optionally, average, minimum, and maximum values for the map can be displayed as thumbnails in the left 60 pixels of the legend.

There are three related scripts: grid_to_map, grid_to_movie, and grid_to_movie_with_filename. All three of these scripts scan immediate subdirectories looking for .csv files and run GridToMap against any they find. They differ only in how they name the output files. grid_to_map simply appends ".bmp" to the new filename. grid_to_movie uses "movie" with a sequence number and ".bmp". grid_to_movie_with_filename prepends "movie", a sequence number, three under-bars, the filename, the sequence number (again) and ".bmp".

Sample grids, pallets, legends, and overlays are included with the download in the GridToMap folder. For additional details, see the version log that comes with the download.

Clouds

2010-03-26T20:01:00.003-04:00

Magicjava Climate Data Announced

2010-03-23T18:30:00.000-04:00

The first beta release of Magicjava Climate Data has been made available. The data is daily and monthly averages for AMSU channel 5 for January, 2009 through December, 2009 in .csv format. You can get the data here.

Aqua Satellite Project, Update 4 Released

2010-03-21T13:31:00.012-04:00

Update 4 for the Aqua Satellite Project is ready. You can download it here. I'd have to say the big item in this release is the QA. But another big item is the new MapToGrid program.

MapToGrid

The MapToGrid program is for converting information in a black and white bitmap image to a 2.5 by 2.5 degree equal area grid that represents the Earth. An example image is shown to the left. It has water drawn in black and land draw in white. MapToGrid can take such information and transform it into C code, Javascript code, XML, or text. Other programs can then use this gridded information as a lookup table.

A screen shoot of a Javascript program that reads grid information and draws the Earth's water and land on an HTML page is shown below.

Javascript HTML image created using Draw2D and MapToGrid

MapToGrid is the first of several programs that, together, will allow the creation and display of gridded satellite data.

The command line help text for the MapToGrid program describes the details of using the program and is provided below.

MapToGrid converts a map image containing only the colors black and white into grid information. The map image must be a 32-bits-per-pixel, 720 by 360 pixel bitmap (.BMP) image. This image will be treated as a map of the entire world. The grid information will be a 2.5 by 2.5 degree equal area grid which indicates the percentage of white pixels in each grid.  The output grid can be formatted as C code, JavaScript code, XML, or text. The default is text. Usage:
MapToGrid [-h] -i Map_Bitmap -o Output_FileName [-f C | JavaScript | XML | text]
h Displays this help text.
i The location the bitmap (.BMP) input image.
o The name of the output file that will store the grid information.
f The output format. Valid format arguments are C, JavaScript, XML, or text.

There are sample bitmaps suitable for using with MapToGrid in the MapToGrid folder.

QA
A QA scrub was given to nearly all the existing programs (AMSUNormalize still needs to be done). Several major bugs were fixed and some new features added.

One of the new features is the the saving off of scans that don't pass NASA QA during an AMSUExtract run. You can take these rejected scans and run them through AMSUSummary just like the regular scans. A screen shot of a graph of rejected scans that has been summarized by AMSUSummary is shown below.

Another new feature is the -FLOOR switch added to AMSUExtract. This option lets you specify a lower value for scans. Any value below the floor is rejected. When using the -FLOOR switch you don't need to specify a value. In this case the value defaults to 150 K (-123.15° C, -189.67° F). The -FLOOR switch replaces the -q switch, which has been removed.

Yet another new feature added due to the QA was AMSUSummary printing out lower third and upper third averages for data rather than minimum and maximum values. This removes a lot of the noise that was created by the minimum and maximum feature. The maximum and minimum features have been removed.

A previous post showed a graph of the lower and upper averages that looked rather blocky. This has been fixed and you can use these features with full confidence.

Corrected Lower Third Averages Example

Change Log
So those are the big items. Here's the text of the change log for full details:

The following improvements were made for Update 4:

*) Added MapToGrid program that converts a map image containing only the colors black and
white into grid information.

The map image must be a 32-bits-per-pixel, 720 by 360 pixel bitmap (.BMP) image. This image
will be treated as a map of the entire world.

The grid information will be a 2.5 by 2.5 degree equal area grid which indicates the
percentage of white pixels in each grid.

The output grid can be formatted as C code, JavaScript code, XML, or text. The default is
text.

Includes command line help text.

*) Updated AMSUExtract program to send data that does not pass NASA's QA to stderr. Updated
amsu_extract script to handle this option. It sends stderr files to a file with the same
name as the regular file, but with the text "_Rejected" just before the file type suffix.

*) Added -FLOOR switch to AMSUExtract. The switch takes an optional value for degrees Kelvin.
Any scan below this value is sent to stderr rather than stdout. The value for -FLOOR defaults
to 150 K (-123.15° C, -189.67° F)

*) Removed -q switch from AMSUExtract. Limb adjustment checks are no longer done by this program.
NASA QA checks are done automatically now, as described above.

*) Default tolerance for Limb Check in AMSUQA raised from 0.5 to 1.0. Channels 12 and 19 added
to the list of channels that no limb check is performed on. Miscellaneous bugs in limb check
values fixed.

*) Modified AMSUExtract to report daily and monthly lower and upper 1/3 averages rather than
minimum and maximum values.

*) QAed AMSUExtract and AMSUSummary, fixing several major bugs.

*) MonthlyData and MonthlyDataMap have been refactored to a more generic ColumnData and
RowDataMap. These new classes are templates. The number of columns, ID type, and data type
are parameterized.

*) Added the ability to retrieve upper and lower averages to ColumnData and RowDataMap
classes.

*) Added avg and sum scripts. These scripts take a file name and a column number and return
the average or sum of that column. These are Unix scrips, requiring Unix utilities like cut,
grep, and awk. These scripts are not and will never be part of the main workflow, they are
intended as debugging aids. These new scripts are located in the Scripts folder. See the
comments inside the scripts for details on using them.

Results Of Aqua Satellite Project QA Checks

2010-03-21T07:35:00.001-04:00

The big QA session for the Aqua Satellite project is done. The results are there were some major bugs in the code. With these bugs fixed, the data from NASA looks they way you'd expect it to. The unusual readings occurring for each footprint mentioned in this post are still there, but they're not enough to throw off the averages.

Major Bug Fixes
We'll take a look at all this data, but first, here's a list of the major bugs that were fixed:

● The Limb Check had values that were too small for some of the footprints in its lookup table. This is what was causing entire footprints to be dropped from the old QA checks.

● A subtle bug in the gnu C++ template generator was generating integers rather than floating point variables when working with floating point data. This was causing the decimal places to be chopped off, throwing off things like average values. This was fixed by getting rid of the offending templates and replacing them with a series of overloaded functions.

● The order in which I was grabbing data for daily and monthly summaries was wrong, causing data for one footprint to be displayed as another footprint. This has been fixed.

● The concept of daily and monthly minimum and maximum values produced by AMSUSummary was and is fundamentally flawed. For reasons why, see my post on Noise. Therefore, the concept has been replaced with averages for the lower third of the data and the upper third of the data. There's enough data that goes into these calculations that noise shouldn't be a problem.

What The Data Looks Like Now
The data displayed here covers the range of January, 2008, and January, 2009 through January, 2010. This is 14 months total of data. Letting the pictures speak for themselves:

Monthly Averages, By Footprint

Monthly Averages, By Date

Monthly Lower Third Averages, By Footprint

Monthly Lower Third Averages, By Date

Monthly Upper Third Averages, By Footprint

Monthly Upper Third Averages, By Date

Daily Averages, By Footprint

Daily Averages, By Date

Daily Lower Third Averages, By Footprint

Daily Lower Third Averages, By Date

Daily Upper Third Averages, By Footprint

Daily Upper Third Averages, By Date

The upper and lower values look a little blocky. So I'll do some more QA on those. I also still need to QA AMSUNormalize. However, none of those things are enough to prevent this code from being released as part of Update 4. Update 4 will be released later today.

Previous Posts In This Series
QAing AMSUExtract
What's Wrong With This Picture?

One Month Of AMSU Channel 5 Data

2010-03-20T05:41:00.001-04:00

This video shows 1 month of Aqua AMSU Channel 5 data as it was displayed during one of my debugging sessions. The video is kinda blurry due to low resolution, but don't worry, you wouldn't be able to read all the data scrolling by even at HD resolution. Enjoy :)

Update:
Here's a screen shot of the end of the video in better resolution so you can see the data is nothing but the numeric readings from the channel.

National Snow And Ice Data Center Releases AMSR-E Satellite Surface Temperature Data

2010-03-19T09:26:00.002-04:00

I just got this in the mail late yesterday. The National Snow And Ice Data Center (NSIDC) has just released several new climate products, including minimum and maximum air surface temperatures.

Here's the e-mail:

NSIDC is pleased to announce the release of the Daily Global Land Surface Parameters Derived from AMSR-E data set. This data set contains satellite-retrieved geophysical parameters generated from the Advanced Microwave Scanning Radiometer - Earth Observing System (AMSR-E) instrument on the National Aeronautics and Space Administration (NASA) Aqua satellite. Parameters include:

-Air temperature minima and maxima at approximately 2 meters in height

-Fractional cover of open water on land

-Vegetation canopy microwave transmittance

-Surface soil moisture at less than or equal to 2 centimeters soil depth

-Integrated water vapor content of the intervening atmosphere for the total column

The daily parameter retrievals extend from 19 June 2002 through 31 December 2008. The global retrievals were derived over land for non-precipitating, non-snow, and non-ice covered conditions. The primary input data were daily AMSR-E dual polarized multi-frequency, ascending and descending overpass brightness temperature data.

For more information regarding this data set, please see http://nsidc.org/data/nsidc-0451.html.

This is pretty timely. There's several simultaneous attempts going on right now to better understand Urban Heat Islands (UHI). The minima/maxima temperatures at 6 meters should be particularly relevant to these studies. A signature of UHI is the night time temperatures go up, while the day time temperatures stay about the same. This new data will give us the tools we need to look for UHI directly on a global scale.

And since the AMSR-E used to gather this data is part of the Aqua satellite, I can fold in studying the new data with the work on the satellite.

So expect to hear more about this in the near future.

QAing AMSUExtract

2010-03-18T20:21:00.018-04:00

In A NutShell
So, I've been QAing AMSUExtract the last couple days. The end results is that several bugs were found and fixed in my code. But even with these fixes, a lot of data was still failing limb validation. Even after increasing the limb validation tolerance from the original 0.5 to 10.0 (giving over 100 degrees tolerance in some cases), there were still more failures than I would consider acceptable.

This told me that having a limb validation in AMSUExtract was pointless. To get a decent number of records to pass this QA check, I'd need to make the check so loose that nearly any imaginable value would pass.

So limb validation, the -q switch of AMSUExtract, has been taken out. It's been replaced with a -FLOOR switch that lets you set a lower bound for a valid temperature. The default lower bound when you use -FLOOR is 150 degrees K (-123.15° C, -189.67° F), but you can pass any numeric value you want. Scans that are marked as bad by NASA or that fail the optional -FLOOR check are the only ones that will now fail in AMSUExtract.

Limb vaidation is still available as an option for the AMSUQA program.

I want to thank Malaga View for suggesting the feature to save not only scans that pass QA, but also scans that fail QA. This feature is now part of AMSUExtract and was very useful for this QA check. BTW, should anyone else have ideas for features, please feel free to suggest them. If I have the time, I'll add them.

The end result of all this is AMSUExtract is now working and has new features to boot. This will be released in Update 4. Now I move on to QAing AMSUSummary.

Final Results With -FLOOR switch.

The file is clean like this all the way through.

The Painful Details

The part of this post can be skipped. It's the details on how the QA was carried out. It's here for people who just love to read about QA procedures on blogs or who may need to know the details for something they're doing on their own.

Setup For QA

I setup the QA runs by removing all but one folder for the directory I was working in. That folder contained .hdf files for a single month. That folder, in turn, contained a text folder that contained the text extract files created by ncdump from the .hdf files.

As always, ncdump was run with no command line arguments, just the file name of the file being converted.

You can download ncdump from here.

Running Release 3 Version Of AMSUExtract

The amsu_extract script was used to extract data from all the files for the month. The exact command line was:

amsu_extract -extension C5F1-30_QA_TEST -c 5 -f 1-30 -q

This extracts channel 5, footprints 1 through 30 and performs QA checks on them. The -extension argument tells the script how to name the output file. In this case the output extract file will be named 200912_extract_C5F1-30_QA_TEST.csv, where 200912 is the name of the folder storing the monthly data. This file is created in the same directory from which the script is run.

The version of AMSUExtract called by the amsu_extract script was the same as the one in Update 3 with one exception: scans that fail QA are sent to stderr. The amsu_extract script, in turn, saves stderr to a separate file.

Results Of Running Release 3 Version Of AMSUExtract

Step 1: Check Values Being Extracted

The first thing to do is verify that AMSUExtract is pulling the correct data. To do this, use HDFView and open up the original .hdf files downloaded from NASA. You can download HDFView from here.

Use the File/Open command to open the .hdf files you want to look at. Use the tree control on the left to navigate to brightness_temp. Double click on brightness_temp. At the to of the grid window that opens up, use the page scroller to move to page 4. Page 4 is the data for channel 5, the numbering starts at zero.

You'll notice the rows in the window are numbered 0-44. These are the scan lines. The columns are numbered 0-29. These are the footprints.

Open up the extract file (200912_extract_C5F1-30_QA_TEST.csv in my case) in a text editor. Visually compare the values in the extract file to the values in HDFView. You don't have to check every value, but you want to check enough cases to convince yourself the correct values are being pulled.

Step 2: Compare File Sizes Of Passed And Failed Data Files

Checking the file sizes of the passed and failed data shows that the failed file is about 2/3s the size of the passed file. Now, a fair portion of that is header information, but most of it is actually failed data. This means a lot of data is failing QA.

Step 3: Look At Failed Data:

Opening up the failed data file, I saw a lot of data that looked reasonable to me. So I was failing data that should have passed.

To fix this problem, I tracked down a couple of bugs in the QA checks, expanded the number of footprints that don't get a limb validation check, and increased the tolerance of the limb validation from 0.5 to 10.0.

Even with these changes, a lot of data was still failing. I'll talk about that in the next section.

Results Of Running Modified Version Of AMSUExtract

To try to cut down on the number of scans failing, I gradually increased the tolerance. Eventually, I had it all the way up to 10.0 and still had more records failing than I expected. Screen shots of running with a 10.0 tolerance are below.

The error file looks a lot better, but further down the file there are still quite a few scans listed as failing.

At this point I realized it was hopeless to try and salvage the concept of running a limb validation in the extract. So I took it out and replaced it with the -FLOOR option.

References:
Download ncdump
Download HDFView

What's Wrong With This Picture?

2010-03-16T17:52:00.001-04:00

Crazy Data From NASA Or Bugs In My Programs?

While I'm waiting to hear from NASA, I've started doing a QA check both of my code and the Aqua Satellite AMSU data. The above picture shows an example of some of the things I've come across. The picture shows NASA's AMSU channel 5 data for every day in January, 2008, and January, 2009 through January, 2010 for all 30 footprints (14 months total). The view along the X axis is by footprint, with an average at the end. The data was generated by my AMSUSummary program and displayed in Apple Numbers.

You can see the obvious problems with the data. Channels 17 and 18 are just whacked. Channels 25 through 30 go in the wrong direction, and channels 1 through 6 have too high a value. Compare this to the more standard picture of what such a footprint snapshot should look like, shown below.

What 30 Footprints Should Look Like.

Below is the same data again, this time the X axis represents time.

And here's daily data from December 31st, 2009. This data has been through my QA process and none of the scans for channels 24 or 25 passed QA. The QA checks to make sure none of the data is more than +/- 50% of the expected readings as defined in the literature.

So the question is, is this strange data due to bugs in my code, or is this the way the data actually looks. I've already done QA checks on my code, but for this next week, I'll be going over it again to see where the problem lies.

So for the rest of the week expect QA posts from me. In addition to checking out the code and data, these posts will give good instructions on how to use the programs I've written so far.

Request Sent To NASA For Unpublished Data And Algorithms Related To Creation Of Synthetic Channel 4 Data

2010-03-15T17:25:00.001-04:00

As discussed in a previous post, NASA now synthesizes channel 4 data of the Aqua AMSU. Going through the steps needed to recreate this synthesis, I noticed several sets of required data are not available to the public. So I've sent off a request for this data as well as the associated (and undocumented, as far as I can tell) AMSU-A Radiative Transfer Algorithm. I've also requested information regarding the depth of the atmosphere each footprint on channel 5 scans so that I can get back to work on recreating the UAH temperatures.

The request was sent a few days ago. Hopefully they'll respond. They've already sent one piece of missing data, the at-launch noise for each channel (that's the NEDTi in the formula at the top of this post). So my thanks to NASA JPL for that.

The requested data is:

● The 230000 cases used to create the values for the vectors Ai and Theta Bar i, or the values of vectors Ai and Theta Bar i themselves if the 230000 readings are no longer available.
● Documentation on how the AMSU-A Radiative Transfer Algorithm actually works.
● Atmospheric scan depth for each footprint on channel 5.

Previous Posts In This Series:

Note On NASA's Lack Of Limb Adjustment

NASA Responds

AMSU Limb Adjustment

Noise

Proof That Temperature Area Determines Temperature Anomaly

Trying To Find The UAH January Anomaly In The Raw Data, Part 1 Of 2

Overview Of The Aqua Satellite Project, Update 1 Features

Aqua Satellite Project, Update 1 Released

Spot Checking The Spot Check

NASA, UAH Notified Of QA Spot Check Findings

About The Aqua Satellite Project

UAH January Raw Data Spot Check

So, About That January UAH Anomaly

A Note On UAH's High January Temperature

References:
NASA Responds
AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4

Aqua Satellite Project, Update 3 Released

2010-03-14T13:41:00.002-04:00

Update 3 for the Aqua Satellite Project is ready. You can download it here. It includes a new AMSUNormalize program for normalizing temperature data produced by AMSUSummary.

Additionally, a -l flag was added to AMSUExtract and AMSUSummary to pull latitude and longitude values for each footprint being pulled.

Those are the big items. For additional details on the release, see the change log.

Draw2D JavaScript Graphics Engine Released

2010-03-13T03:07:00.001-05:00

Draw2D version 1.0 has just been released and you can get it here.

Draw2D is a JavaScript graphics engine that allows you to add dynamically created shapes to any web page. Draw2D also supports mouse events, allowing shapes to respond to user input. Like all my coding projects, the Draw2D code has been placed in the public domain.

Draw2D will serve as the graphics engine for new features available in the Climate Scientist Starter Kit, Version 2.0. More details on this will be given in future posts.

The download includes source code, several HTML example pages, and a programmer's guide in Apple Pages, Microsoft Word, and Adobe PDF formats.

I'll finish out this post with a few screen shots showing what Draw2D can do.

Points, Lines, Quadradic Curves, Cubic Curves

Triangles And Polygons

Rectangles And Round Rectangles

Ovals And Arcs

The Case For Raw Data And Source Code

2010-03-12T18:49:00.000-05:00

There's been a bit of chatter about raw data and source code in science on a few of the climate blogs lately. The issue has tended to be framed as "raw data" vs "anomalies". I think that's a fairly poor way to frame it because anyone using anomalies can also use raw data, and vise versa.

Bottom line, I think it's extremely important for scientists to make all their raw data and computer source code available to the public. The rest of this post will present a few reasons why.

The Computer Is The Science
Computers increasingly play a central role in science. They don't just tabulate results and draw graphs. They are actually the tool used to perform measurements and experiments and to even define the physics used to conduct the experiments.

String Theory, for example, has very little relationship to the real world from an experimental point of view. Most of String Theory is code running on a computer. Climate models too are computer programs that define a physics and execute entirely on computer.

I'm not trying to make an argument over whether or not doing science on a computer is good or bad. I'm merely pointing out that it's done, and it's rather common.

Point 1: Verifiability
A consequence of having your physics defined by code and your universe defined by data is that the experiments run on a computer cannot be verified without the code and data. In order to have verifiability, a fundamental goal of science, the code and data must be available. I think this is obvious, but I want to present an example of unverifiable science that was just reveled on this blog.

The Aqua satellite has an instrument called an AMSU that scans the Earth detecting 15 different frequencies of light. Each of these detectors is called a channel. It turns out that the hardware for channel 4 failed in late 2007, and since that time NASA, who owns the Aqua satellite, has been producing the data from channel 4 not from readings detected by instruments, but from computer code and lookup data.

They actually have good reasons on why this should work. They've performed tests using their code and lookup data and the results match reality very well. The problem is, the code and lookup data to create this synthetic channel 4 data is not available outside NASA's JPL lab. This means that not only has no one from outside JPL ever verified their claims that the procedure is sound, no one outside JPL can verify those claims. We (no mater who "we" are) have to take it on good faith that the data NASA creates for channel 4 is realistic. It cannot be demonstrated to be realistic.

Clearly, this is not science. To get science back on the track towards verifiability in the modern world, open access to all data and computer code is needed.

Point 2: Data Compatibility
I think this second point applies more to climate science in particular than science as a whole. It has to do with the concept of processing temperature data into anomalies. There are very good reasons why scientists convert temperature readings into anomalies. If done with care, anomalies can be used to meaningfully compare two different sets of data, for example.

Anomalies also have the nice feature of tracking changes in temperature in a way that's not dependent on the temperature that came just before it. For example, a July anomaly of, say, +0.2 doesn't mean that July was +0.2 degrees hotter than the previous June. It means it was +0.2 degrees hotter than the July from the previous year. All by itself, this use of anomalies filters out unwanted noise from that data. We're not interested in learning if it gets hotter as we move into summer. We already know it usually does. We're interested in learning if the trend over the years is going up or down. And that's what anomalies tell us.

But it's difficult to compare data in anomaly form to data that's not in anomaly form. And most climate data, CO2, water vapor, sun spots, cosmic rays, etc., etc. is not in anomaly form. So if we have a July that's cooler than the previous July but warmer than the June just before it, the July anomaly goes down, where as the July temperature and most other climate data that's correlated to temperature, goes up.

Point 3: Errors
I'll make this last point quick, because this post is longer than I expected.

You simply cannot find errors in data that's been homogenized and processed until it no longer has it's original shape. Smoothing, averaging, and so on, wash away these errors and the person using the data doesn't know they were ever there. This doesn't mean, btw, that such processes correct the errors. They simply hide them. To verify that the data is correct, both the original data and the computer code used to process it is required.

Conclusion
So these are some of the reasons I believe it's important to make all data and computer code available to the public. I want to stress again that making raw data and code available is not an argument against using the processed data results and no code. There's no reason why you can't do both.

Climate Scientist Starter Kit v2.0 Coming, Part IV: Total Solar Irradiance, Sunspots, Regional Cosmic Rays

2010-03-11T02:28:00.000-05:00

In this post we look at several new data sets being added to version 2.0 of the Climate Scientist Starter Kit: Total Solar Irradiance (TSI), Sunspots, and regional Cosmic Rays.

Total Solar Irradiance (TSI)
Total Solar Irradiance is the amount of sunshine hitting the Earth's atmosphere at any given time. Version 2.0 of the Climate Scientist Starter Kit includes monthly TSI values taken from three satellites at three different frequencies. This data extends from February, 1996 to October, 2009.

Total Solar Irradiance (TSI)

Sunspots

Version 1.5 of the Climate Scientist Starter Kit already has Sunspot Group data. Version 2.0 adds Sunspots. This data extends from January, 1749 to January, 2010.

Sunspots

Regional Cosmic Rays
Regional Cosmic Rays have been added to the existing Cosmic Ray data of version 1.5. With this added data, there is now at least one cosmic ray measurement for every region used by UAH when it publishes temperatures: Northern Hemisphere, Southern Hemisphere, Tropics, Northern Extra Tropics, Southern Extra Tropics, North Pole, South Pole, and the 48 U.S. continental states.

Thule, Greenland (North Pole Region)

Swarthmore, Pennsylvania (U.S. 48 States, Northern Extra Tropics)

Climax, Colorado (U.S. 48 States, Northern Extra Tropics)

Haleakala, Hi (Tropics)

Hauncayo, Peru (Tropics)

Hermanus, South Africa (Southern Extra Tropics)

McMurdo, Antarctica (South Pole)

South Pole (South Pole)

McMurdo, Swarthmore, South Pole, Thule, Cosmic Rays

Previous Posts In This Series:
Climate Scientist Starter Kit v2.0 Coming
Climate Scientist Starter Kit v2.0 Coming, Part II: Regional Data
Climate Scientist Starter Kit v2.0 Coming, Part III: Ozone And Pressure

Note On NASA's Lack Of Limb Adjustment

2010-03-10T07:37:00.003-05:00

I talked about limb adjustment in this post. In part of NASA's reply to my questions about their data, they noted they don't do limb adjustments to the data. Specifically, they said:

The AIRS retrieval code (statistics engine) does not incorporate a limb adjustment as you have described above. Where reliable sensor data is available, it is applied directly to the appropriate portion of the atmosphere, taking into account the angle of the observation.

So unlike NOAA who adjusts the readings to have them all at the same altitude, NASA uses the different readings at different footprints for exactly what they are: measurements at different levels of the atmosphere.

This means that I have to make sure the footprints I'm using are the footprints used by UAH. It may turn out that footprint 15 of the channel 5 data, which I used in this previous post, is not used by UAH. I may need to look further out to get the data UAH uses. So when I finally get to part two of the UAH anomaly post, I'll make sure Im using the correct footprints.

References:
AMSU Limb Adjustment
NASA Responds
Trying To Find The UAH January Anomaly In The Raw Data, Part 1 Of 2

Previous Posts In This Series:

NASA Responds

AMSU Limb Adjustment

Noise

Proof That Temperature Area Determines Temperature Anomaly

Trying To Find The UAH January Anomaly In The Raw Data, Part 1 Of 2

Overview Of The Aqua Satellite Project, Update 1 Features

Aqua Satellite Project, Update 1 Released

Spot Checking The Spot Check

NASA, UAH Notified Of QA Spot Check Findings

About The Aqua Satellite Project

UAH January Raw Data Spot Check

So, About That January UAH Anomaly

A Note On UAH's High January Temperature

NASA Responds

2010-03-09T02:40:00.005-05:00

UPDATE:
Re-reading their response, I noticed they didn't actually admit in the body of the e-mail to making up all of channel 4's readings. But that's what they do. You can read about it at one of the links they provided here: AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4
===

I just got a response from NASA about the various questions I raised in this post regarding the AMSU on the Aqua satellite.

Basically, their response is that channel 4 failed sometime in late 2007 and now they invent the readings from channel 4 from whole cloth and feed those invented readings into the calculations.

Despite my post on noise, this invention of an entire channel's readings isn't as far-featched as it first sounds. The recurring patterns that exist in all the footprints for all the channels make this far easier to do than if we were dealing with something like land-based thermometer readings. For example, given a single footprint reading on any channel, you can make a reasonable guess as to what all the other footprint readings will be for that channel. The between-channel values also have a very consistent relationship to each other. An example of that consistent relationship is shown in the graphic at the beginning of this post.

They also said that having no automaticQualityFlags for a month marked as "passed" is not a big deal, and that the values for badData, etc., are not the same in every file. On that last one, I'm not so sure. I dug into the files with a text editor and looked at the badData values myself, rather than just trusting my code. They were all the same. But, for now, I'll take their word on it and assume there's something there that I'm not yet understanding.

Anyway, here's their response:

Thank you for your interest in Aqua AMSU-A data. Before your specific questions are answered, understanding the following background information about Aqua AMSU-A will be helpful. All documents referenced here can be found at our public archive at the Goddard Earth Sciences Data and Information Services Center (GES DISC) at http://disc.sci.gsfc.nasa.gov/AIRS/documentation.

Aqua AMSU-A is a microwave sounder that is very similar to AMSU-A instruments flown on many NOAA satellites as well as on the European MetOp-A satellite. Aqua AMSU-A senses the atmosphere in 15 distinct data channels. One of these channels, Channel 4, failed in late 2007, as described in AMSU-A Channel 4 NeDT Update: 20 December 2007 at the document archive.

Aqua AMSU-A products include a variety of flags to indicate data quality. However, not all data quality flags are particularly useful for indicating actual data quality. In particular, ‘AutomaticQualityFlag’ is not an effective flag to check because any datasets less than 100% complete will be marked “suspect.” You may wish to review the AIRS Version 5 Released Files Description document to find more suitable QA flags. We suggest you look at “NeDT,” “state1” and “state2” as being better data quality indicators.

Since the degradation of AMSU-04, the AIRS Level 2 retrieval code does not utilize AMSU channel 4 brightness temperatures. You may want to review AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4 to understand how we are addressing the loss of Channel 4 within our retrieval software.

With that background, let us address your 3 specific questions:

Is it considered normal to have zero Level 1B AMSU data files for a month pass QA?

It is a mistake to characterize a granule in which AutomaticQualityFlag is set to "Suspect" as not passing QA. In fact, this is a “normal” condition since late 2007, and it merely reminds us that the dataset is not 100% complete.

Is it normal for all Level 1B AMSU data files for a month to have the exact same numbers for bad data, missing data, special data, and total data?

All Level-1B data files for the month of January 2010 do not have identical values for NumBadData, etc. For example, data collected over a spacecraft maneuver will reflect the state of the instrument at that time. This situation is typical, and most data for any given month should be in the current nominal state. The typical month will contain 1-3 short intervals of bad data from spacecraft maneuvers.

Doesn't the statistics engine used for AMSU limb adjustment require valid data from channel 4 in order to correctly adjust channel 5 data, the channel which is used to create temperature anomalies provided to the public?

The AIRS retrieval code (statistics engine) does not incorporate a limb adjustment as you have described above. Where reliable sensor data is available, it is applied directly to the appropriate portion of the atmosphere, taking into account the angle of the observation. Of the 2378 infrared and 15 microwave channels available to the AIRS retrieval algorithm, no particular channel is most important in deriving our products. Instead, the unique combination of all these channels of data allows us to develop a very complete and accurate temperature and water vapor profile throughout the entire atmosphere, and that is why our data products are very important to weather forecasting and climate studies.

Again, the AIRS Project believes that many of your technical questions can be answered reviewing the documentation at: http://disc.sci.gsfc.nasa.gov/AIRS/documentation. If you have further questions, we request that you contact us via our “Ask AIRS” portal at http://airs.jpl.nasa.gov/AskAirs/. You may also want to register as an AIRS Data User at http://airs.jpl.nasa.gov/DataRegistration/data/index.cfm. In that way you will be notified whenever a significant announcement regarding the AIRS Project or the AIRS and AMSU-A instruments is issued.

Thank you for your interest in AIRS data,

Previous Posts In This Series:

AMSU Limb Adjustment

Noise

Proof That Temperature Area Determines Temperature Anomaly

Trying To Find The UAH January Anomaly In The Raw Data, Part 1 Of 2

Overview Of The Aqua Satellite Project, Update 1 Features

Aqua Satellite Project, Update 1 Released

Spot Checking The Spot Check

NASA, UAH Notified Of QA Spot Check Findings

About The Aqua Satellite Project

UAH January Raw Data Spot Check

So, About That January UAH Anomaly

A Note On UAH's High January Temperature

References:

NASA, UAH Notified Of QA Spot Check Findings

Noise

Aqua Satellite Project, Update 2 Released

2010-03-07T15:14:00.003-05:00

Update 2 for the Aqua Satellite Project is ready. You can download it here. It includes a new AMSUSummary program for summarizing temperature data into the usual daily and monthly values. AMSUSummary provides average, minimum, and maximum values by day and month.

Like last weekend, the all night coding sessions have left me exhausted, so the change log will have to do for now as a description of the new features.

=== UPDATE 2 March, 7, 2010
The following improvements were made for Update 2:

*) Added AMSUSummary program for summarizing extract data in csv format. Summaries include daily and monthly averages, minimums, and maximums. Includes command line help text.

*) Bash shell script for running AMSUSummary has been added. It's name is amsu_summary and it's located in the Scripts folder. See the comments inside the script for details on using it.

*) Added Limb Effect Check to AMSUQA. This check is turned on using the -l switch. The limb effect check examines channels 4, 5, and 6 to ensure their readings are within expected limb variations as defined in "The Limb Adjustment of AMSU-A Observations: Methodology and Validation" Goldberg, et al. 2001. Footprints 1 through 12 and 19 through 30 are checked. Footprints 13 through 18 are not checked due to their small limb effect values, which allows for even small fluctuations to cause a failure.

The -l switch has an optional tolerance parameter which defaults to 0.5. This parameter specifies how much above or below the values of Goldberg, et al. 2001 a reading can be. A value of 0.5 provides a margin of +/- 50%. A value of 1.0 provides a margin of +/- 100%.

*) Changed AMSUQA -f switch to -i. Added -p switch to specify output file name prefix. amsu_qa script changed to use new switches.

*) Changed AMSUExtract -f switch to -i and -F flag to -f. amsu_extract script changed to use new flags.

*) Added -q switch to AMSUExtract to indicate only data that passes QA should be extracted. The -q switch includes a check for limb effect at 0.5 tolerance. Only data QA values are checked. File-wide QA values are ignored, allowing data to be extracted from files that have not passed QA.

*) Modified amsu_qa Bash shell script. Output is now placed in the directory from which the script is run, to bring it in line within how amsu_extract and amsu_summary work. See the comments inside the script for details on using it.

*) Bash shell script for removing HDF text files has been added. It's name is rm_hdf_text and it's located in the Scripts folder. See the comments inside the script for details on using it.

*) Added Doxygen files for the source code. This is in the Doc/Doxygen/html folder.

*) Added Aqua Satellite PDF book. This is in the Doc folder.

*) Added HTML pages describing the commands and scripts.This is n the Doc/HTML folder.

*) Added zip file for sample AMSU data for October 1st, 2009. This is in the Sample Data folder.

*) Added CodeBlocks projects for cross-platform build support. These are in the CodeBlocks folder.

AMSU Limb Adjustment

2010-03-04T23:15:00.001-05:00

This post looks at the Limb Adjustment done for the AMSU. We talk about what the Limb Adjustment is, how it is done, and some problems associated with it.

What Is The Limb Adjustment?
If you stand up, bend over at the waist and swing your arm in an arc under you, you'll notice your arm is closest to the floor when it's directly under you. And you'll notice your arm gets further from the floor as it continues through the arc in either the left or right direction.

The scanning of the AMSU on the Aqua satellite has a similar situation. When it scans directly below itself, it gets data from closest to the surface, but the scans to the left and right of the satellite don't penetrate as deeply. Because of this, the temperatures read by the scans to the left and right need to be adjusted. This is called a Limb Adjustment. Publishing in the American Meteorological Society, Quanhua Liu and Fuzhong Weng (2006) had this to say about Limb Adjustment:

A remarkable effect of the cross-scan sensor is the variation of the brightness temperatures across the scan line, even though the scene temperature is homogeneous. The variation in the cross-track measurements due to the change of the scanning angle is called limb effect and can be as much as 30 K for the 23.8-GHz water vapor channel and 15 K for troposphere sounding channels (Goldberg et al. 2001). Because the limb effect is often stronger than the real variation of the signatures from scenes, the unadjusted measurements prevent the objective analysis of weather systems and may make the regression retrieval algorithm complicated. More important, averaging satellite brightness temperatures to a given grid map for climate study requires that the data be limb adjusted prior to averaging.

For microwave instruments, like the AMSU, a different form of adjustment needs to be done over land and water, due to surface emissivity. Again from Liu and Weng (2006):

It is a little complicated for the microwave channels. The surface emits either more or less than the atmosphere at the microwave range, with full dependence on the surface emissivity. The water surface may emit much less energy than the atmosphere in the microwave range. The weighting function of the microwave troposphere channel is broader than that of the infrared channel. Asymmetric behavior of AMSU-A channels on the two sides of the nadir is recognizable (Weng et al. 2003).Goldberg et al. (2001) have developed a limb-correction algorithm to overcome the difficulties for AMSU-A. They computed the limb adjustment from multiple-channel observations and the scan position–dependent coefficients. Their algorithm is routinely applied for National Oceanic and Atmospheric Administration (NOAA) operational products.

How Limb Adjustment Is Done
A collection of scans from the month of July, 1998 is used to provide a mean for each latitude for each footprint, within 2˚ latitude. These historical scans are combined with current scans from the channel being examined and its neighboring channels, and a set of physical and statistical coefficients. Publishing in the American Meteorological Society, Mitchell D. Goldberg, David S. Crosby, and Lihang Zhou (2001) had this to say about Limb Adjustment:

A global set of coefficients is used for channels 6–14. Separate sea and nonsea coefficients are used for channels affected by the surface—channels 1–5 and 15. The predictors are generally the channel itself plus the adjacent channel whose weighting functions peak below and above. In other words to limb adjust channel 6, we use unadjusted channels 5, 6, and 7 observations as predictors. The exceptions are channel 14 uses channels 12, 13, and 14; channel 3 uses channels 3, 4, and 5; channel 1 and 2 both use channels 1 and 2, and channel 15 uses channels 1 and 15.

So to adjust for, say, channel 5, the historical values for channel 5 at the satellites current location, the current scan values for channels 4, 5, and 6, and a set of physical and statistical coefficients are used.

Potential Problems With Limb Adjustment

From what I can see, there are two potential problems with Limb Adjustment. The first is when current scan values are outside the limits expected based on the historical scans. For example, the scan line shown at the beginning of this post has a value at foot print 1 that is 40% below the expected low limit, which is shown in the diagram at the start of this section.

The second potential problem is when a neighboring scan line used for the adjustments doesn't have any available data. For channel 5, the channel we've been looking at in this series of posts, the neighboring scan lines are 4 and 6. Channel 4 had no data at all in it for the entire month of January. A sample of this is shown in using HDFView in the image provided below.

Screen Shoot Showing No Data In Channel 4.

Click for larger image.

Previous Posts In This Series:
Noise
Proof That Temperature Area Determines Temperature Anomaly
Trying To Find The UAH January Anomaly In The Raw Data, Part 1 Of 2
Overview Of The Aqua Satellite Project, Update 1 Features
Aqua Satellite Project, Update 1 Released
Spot Checking The Spot Check
NASA, UAH Notified Of QA Spot Check Findings
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

References:
Uses of NOAA-16 and -18 Satellite Measurements for Verifying the Limb-Correction Algorithm
The Limb Adjustment of AMSU-A Observations: Methodology and Validation

Noise

2010-03-03T19:56:00.005-05:00

Before doing Part II on looking for the UAH January anomaly in the raw AMSU data, I figured I'd clear the boards regarding "stuff you should know when looking at data". This post covers noise. There's one other post, the AMSU Limb Adjustment, which I'll do after this one.

Noise
There's actually several definitions of noise. In this post we're going to go with noise being anything that significantly changes the signal we're interested in.

You can get noise in a signal by adding additional data to it that's not related to the original signal, which I think everyone knows. But you can also get noise in a signal by taking out data that is related to the original signal. That's not as obvious and it's why I added the graphic showing loss of information degrading a signal into noise.

Creating noise both by adding data and by taking out data is related to my post examining channel 5, footprint 15. In that post, I tried to find validity for UAH's large January, 2010 anomaly by looking at channel 5, footprint 15. That footprint and footprint 16 require almost no limb adjustments to their readings. What they read is very close to the actual temperature.

This is the reason I picked that channel and footprint. It has the smallest chance of adding noise due to any invalid statistical limb adjustments.

The drawback to using only that footprint is I could have very easily created noise by limiting myself to only that footprint. There may be important information in the other footprints that was overlooked.

I'm aware of this problem and it's why we're going to continue our search for evidence of the January anomaly in the raw data. But I thought it was important for readers to be aware of it too.

Previous Posts In This Series:
Proof That Temperature Area Determines Temperature Anomaly
Trying To Find The UAH January Anomaly In The Raw Data, Part 1 Of 2
Overview Of The Aqua Satellite Project, Update 1 Features
Aqua Satellite Project, Update 1 Released
Spot Checking The Spot Check
NASA, UAH Notified Of QA Spot Check Findings
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

Proof That Temperature Area Determines Temperature Anomaly

2010-03-02T05:43:00.006-05:00

In the previous post I used some geometry to show that the January UAH anomaly of 0.72˚ is not supported by the data on channel 5, footprint 15. The idea that the area of a temperature graph can determine the anomaly may not be immediately obvious, so I wanted to show that this must be so with an informal proof. I leave formalizing the proof as an exercise to the reader.

It's important this be done because in the raw data we're working with temperatures, not anomalies. But the published temperature changes are in anomalies, not temperatures. We need to be able to switch between the two and compare the two with confidence.

What Is An Anomaly?
By definition, an anomaly is a deviation from some standard. A temperature anomaly is a deviation from some standard temperature. The value of the standard temperature doesn't matter too much, so long as it's well defined. We'll say the standard temperature has some value we'll call ⣿. Like any other temperature, we can graph ⣿ on a chart. We'll say the temperature ⣿ is graphed by the rectangle a in the diagram at the beginning of this post. When we say the temperature anomaly is 0˚, that's equivalent to saying the temperature is ⣿.

What Is The Temperature Area?
The temperature area is simply the area of the graph below the curve showing the anomaly. So if the anomaly is zero, the temperature area is the area of the bar that maps ⣿.

The X axis of such a graph represents time. When can measure time in different ways. For example the X axis of a month with 31 days can be measured with 333,450 scan lines along the X axis. Or it can be measured as 31 days. Or it can be measure as 1 month. It doesn't really mater which unit we choose anymore than it matters if we call 36 inches 3 feet or a yard.

For convenience, we'll say the X axis is 1, for 1 month. (Note that I used 31 as the width in my previous post. That works just fine, but I'll probably use 1 from here on out to make it easy on myself.) This means when we want to calculate a temperature area, we need only be concerned with it's height. This is because the area of a rectangle equals width times height and width is always 1.

So, for example, the temperature area of ⣿ is always ⣿.

Graphing Anomalies
We can graph anomalies individually, or we can take the best fit to the anomalies. When we take the best fit we get a triangle above or below ⣿. If we instead graph anomalies we get lots of little triangles along the graph, but in the end the area of the anomalies is the same. Again, for convenience, we'll take the best fit and work with a single triangle.

If the anomaly at the beginning of the graph is zero and at the end of the graph is some positive number, ❢, then a triangle is formed with an area equal to ½❢ and the temperature area now equals ⣿ + ½❢. This is shown in rectangle b in the diagram at the beginning of this post.

If the anomaly at the beginning and end of the graph is some positive number ❢, then two triangles are formed both with an area equal to ½❢ and the temperature area now equals ⣿ + ❢. This is shown in rectangle c in the diagram at the beginning of this post.

If the anomaly at the beginning of the graph is zero and at the end of the graph is some negative number, ❧, then a triangle is formed with an area equal to ½❧ and the temperature area now equals ⣿ - ½❧. This is shown in rectangle d in the diagram at the beginning of this post.

Conclusion
We can see from these examples that as the anomaly changes, the temperature area also changes in direct proportion. Or, equivalently, we could say that as the temperature area changes the anomaly changes in direct proportion.

So if we ever see a change in temperature area and a change in anomaly not in sync, we know something is wrong.

Previous Posts In This Series:
Trying To Find The UAH January Anomaly In The Raw Data, Part 1 Of 2
Overview Of The Aqua Satellite Project, Update 1 Features
Aqua Satellite Project, Update 1 Released
Spot Checking The Spot Check
NASA, UAH Notified Of QA Spot Check Findings
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

Trying To Find The UAH January Anomaly In The Raw Data, Part 1 Of 2

2010-03-01T16:29:00.019-05:00

This is a two part series where we look to see if the January UAH anomaly can be found in the Aqua AMSU Level 1B data. The size of the anomaly we're looking for is 0.7˚ C. Each of the two parts of this post will use a slightly different method to try and find the anomaly.

I'll start off right away and tell you the answer for this post: it's not there. The rest of the post will show why.

Calibration With Past Anomalies, Footprint 15.
Both posts will use a method of calibrating the raw temperatures of the AMSU to published UAH anomalies for the same time period. In this post we'll look at footprint 15. We're using footprint 15 because it requires almost no statistical adjustment my the software for limb correction, so we don't have to worry about limb correction, bad data in channel 4, etc.

Calibration is done using 2 periods of published UAH anomalies. In this case, the January, 2008 and January, 2009 data are our calibration periods. We will then use that calibration information to examine the third period, January, 2010. The anomalies for all three periods were reported as:

January, 2008: -0.05
January, 2009: +0.3 (+0.35 from previous year)
January, 2010: +0.72 (+0.42 from previous year)

The raw data for channel 5, footprint 15 for those three months is shown below. All 1,000,350 scans for channel 5, footprint 15 are shown for the three months we're looking at. A trend line is added for each and we zoom in on the beginning and end of the series. January, 2010 is shown in red, January, 2009 is shown in purple, and January, 2008 is shown in green.

January 2008, 2009, 2010 Channel 5 , Footprint 15 Raw Data

Click for larger image

Let's look at the 2008 and 2009 data first. The beginning and ending values are:

January, 2008, Beginning: 246 K
January, 2008, End: 248.75 K
January, 2009, Beginning: 248.2 K
January, 2009, End: 249

We need a way to relate the raw temperature data to the temperature anomalies. Both raw data trends are straight, so we can treat each trend as a right-angled trapizium (i.e. a rectangle with one edge not parallel) with edges at the left, right, bottom, and trend line. This, in turn, can be divided into a rectangle and a triangle. An example of this is shown below for the January, 2008 trend.

From this we can calculate the area of the the first two trends and compare their difference to the difference in anomalies over the same period. This gives us an objective scale between the area of the raw data and the anomaly for the month.

January, 2008 Area: 7668.625
January, 2009 Area: 7706.6
Difference in Area (2009 - 2008): 37.975
Difference in Anomaly (2009 - 2008): +0.35
Anomaly/Area Ratio: 0.0092

So a difference in area of 37.975 increased the anomaly by +0.35 K. Now lets do the same area calculation for 2010 and see what the relationship is.

January, 2010, Beginning: 249 K

January, 2010, End: 248.75

January, 2010 Area: 7715.125

Difference in Area (2010 - 2009): 8.525

Difference in Anomaly (2010 - 2009): +0.42
Anomaly/Area Ratio: 0.0492

The change in area between 2010 and 2009 is less than ¼ the change between 2009 and 2008, yet change in anomaly between 2010 and 2009 is larger than the change in anomaly between 2009 and 2008. The differences in the effect of increasing the temperature area on the anomaly is captured in the Anomaly/Area Ratio. The ratio for the January, 2010 - January, 2009 change is much larger (0.0492) than the ratio for the January, 2009 - January, 2008 change (0.0092).

This clearly can't be right. In these examples, there's no relationship between an expanding temperature area and an anomaly. There should be, because an increase in anomaly represents nothing more than an increase in temperature area.

And since the rules of geometry aren't going to change, it seems there's nothing in this channel 5, footprint 15 data to support the anomaly of January, 2010.

Edit: I used 31 as the width here, representing 31 days. In the future, I'll probably use a width of 1, representing 1 month. This'll simplify calculations.

So Does This Prove The UAH January, 2010 Anomaly Is Wrong?

No. It only shows there's nothing in channel 5, footprint 15 to support the value of the anomaly as valid. Channel 5, footprint 16 will give similar results.

Those two footprints are the ones that really don't need any statistical limb adjustment. The other footprints do. So that means the support for the January, 2010 anomaly either doesn't exist at all, or exists somewhere in those other footprints with their statistical adjustments.

Which is what we'll look at next post. I don't know what we'll find when we look there. I haven't done the analysis yet.

Previous Posts In This Series:

Overview Of The Aqua Satellite Project, Update 1 Features
Aqua Satellite Project, Update 1 Released
Spot Checking The Spot Check
NASA, UAH Notified Of QA Spot Check Findings
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

References:

Dr. Spencer's January UAH Anomaly Post

Overview Of The Aqua Satellite Project, Update 1 Features

2010-03-01T11:20:00.009-05:00

In this post we're going to take a look at the three new shell scripts and new executable that come in Update 1 of the Aqua Satellite Project.

New Scripts
All three of the new shell scripts are for working with HDF files en masse. They all expect a directory structure similar to the one shown in the picture above. In this structure there are a series of folders that all contain HDF files.

The hdf_to_text script will walk this structure looking for HDF binary (.hdf) files in the immediate subdirectories. When it finds them, it creates a text subfolder in the folder containing the HDF files, converts the HDF files to text and places them in the text subfolder. At the end of the script, each of the folders shown above would have a text subfolder containing text versions of their HDF files.

The amsu_qa script is similar. It walks the subfolders and converts .hdf binaries to text in a text folder just like hdf_to_text does, and then it runs AMSUQA on the resulting text file. This saves you the trouble of having to run both scripts. The amsu_qa script is also smart enough not to convert an .hdf file to text if a text version already exists. The output of the amsu_qa script is up to three files in the text folder. These files are named passed, failed, and suspect, and contain the QA results for the files that NASA's QA marked as passed, failed, or suspect in the automaticQualityFlag of the file.

The amsu_extract script is just like the amsu_qa script, except that it runs the new AMSUExtract program rather than AMSUQA. AMSUExtract allows you to provide several different command line switches to get the data you want, and only the data you want. The amsu_extract script lets you specify these switches and it passes them on to AMSUExtract. Generally, you'll be interested in the -s, -c, and -F switches, which allow you to specify the scan lines, channels, and footprints that are extracted.

AMSUExtract
The AMSUExtract program allows you to pull scan information from a Level 1B AMSU text file. Output is in csv format and is printed to standard output, allowing you to redirect it as needed. AMSUExtract has a number of command line switches that control its functionality. These are shown below as displayed by the command's help text.

AMSUExtract Extracts scan information from an AMSU text file and sends results to standard output. Usage:
AMSUExtract [-h] -f AMSU_Text_File [-s Scanlines] [-c Channels] [-F footprints] [-a] [-H]
f The location of the AMSU text file to be QAed.
s An optional comma seperated list of scanlines to extract. Ranges can be specified using a dash. Example -s 1-5,7,10
c An optional comma seperated list of channels to extract. Ranges can be specified using a dash. Example -c 1-5,7,10
F An optional comma seperated list of footprints to extract. Ranges can be specified using a dash. Example -F 1-5,7,10
a Use this switch to extract antenna temperatures. The default is to extract brightness temperatures.
H Do not display headers. The default is to display headers.

The -f switch is required. It specifies the location of the HDF text file to process.

The -s, -c, and -F switches allow you to specify the scan lines, channels, and footprints to extract. You can specify a single number, a collection of comma separated numbers, or a range of numbers with a dash in between the start and end of the range. The comma separated numbers and ranges can be used together. The command to pull all 45 scan lines of channel 5 with footprints 15 and 16 looks like this:

AMSUExtract -f filename -c 5 -F 15,16

The first few lines of the results of that command look like this:

Beginning Date,Clock,Scan Line,Channel 5 Footprint 15 Brightness,Channel 5 Footprint 16 Brightness
2009-12-31,536457572,1,260.328,260.598
2009-12-31,536457580,2,260.262,260.116
2009-12-31,536457588,3,259.891,259.744

The output shows the date, clock time, and scan line number, followed by the channel and footprint values requested. Note that the -s, -c, and -F switches are all optional. If you don't use them, you'll extract all scan lines, all channels, and all footprints.

The csv output format of AMSUExtract makes it easy to build graphs or load values into spreadsheets. The image below shows a graph of scans for channel 5, footprint 15 for all 333,450 scan lines of January, 2010. The data was produced by the amsu_extract script and graphed using an application named DataGraph.

Requirements
All the scripts and programs work on text versions of HDF files. To convert the binary HDF files from NASA into text, use ncdump. To get ncdump, follow this instructions in this post.

The code itself is written in standard C++ and built using the free gcc compiler. It has only been tested on a Macintosh, but in theory should compile and run on any system.

The bash scripts naturally require the free bash shell.

Previous Posts In This Series:
Aqua Satellite Project, Update 1 Released
Spot Checking The Spot Check
NASA, UAH Notified Of QA Spot Check Findings
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

See Also:
AMSR-E And AMSU HDF-EOS C++ Readers Are Done

References:
Looking At The Aqua Satellite Data
DataGraph 2.1.1
gcc compiler
bash shell

Aqua Satellite Project, Update 1 Released

2010-02-28T18:23:00.008-05:00

After a hard weekend of coding, Update 1 for the Aqua Satellite Project is ready. You can download it here. It includes a new AMSUExtract program for pulling temperature data.

Future posts will take a look at the functionality provided by the update, but for now here's the change log entry on what's been added:

=== UPDATE 1 February, 28, 2010
The following improvements were made for Update 1:

*) Speed improvements. The HDFReader class got a significant boost in speed by adding internal indexing for all variable access.

*) AMSUQA output modified to csv format for easy viewing in spreadsheets.

*) Help text added to AMSUQA command line options.

*) Added AMSUExtract program for extracting temperature data in csv format. Includes command line help text.

*) Code cleanup. Helper functions such as split() where previously repeated in every .cpp file that needed them. The standard method of putting such functions in headers is now used. The placement of code in folders has been cleaned up as well.

*) HDFNode and HDFNodeDocument have been refactored to a more generic Node and NodeDocument. These new classes are templates. The data type and attribute data type are parameterized. Two typedefs are provided for string data and attributes. These typedefs are named StringNode and StringNodeDocument.

*) NodeDocument query syntax has been modified to make it more in line with XPath syntax. A separator character must now appear at the beginning of a query string. All existing code that was using queries has been updated to match the new syntax.

*) Bash shell script for running AMSUQA has been added. It's name is amsu_qa and it's located in the Scripts folder. This script will convert compressed binary HDF files to text as needed. See the comments inside the script for details on using it.

*) Bash shell script for running AMSUExtract has been added. It's name is amsu_extract and it's located in the Scripts folder. This script will convert compressed binary HDF files to text as needed. See the comments inside the script for details on using it.

*) Bash shell script for converting compressed binary HDF files to text has been added. It's name is hdf_to_text and it's located in the Scripts folder. See the comments inside the script for details on using it.

*) Macintosh XCode project for AMSUQA and AMSUExtract added. The build folders of these projects includes binaries that will run on a 64-bit enabled intel Macintosh.

*) Added this change log.

Previous Articles In This Series:
Spot Checking The Spot Check
NASA, UAH Notified Of QA Spot Check Findings
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

References:
Aqua Satellite Project (BETA) Update 1

Spot Checking The Spot Check

2010-02-26T02:18:00.003-05:00

Before I posted the results of the QA spot check described here, I first checked those results to make sure they were accurate. This post describes how I checked them.

Checking Binary Data

The Aqua Satellite AMSU Level 1B data comes in compressed binary form. You can view the data using HDFView, which is a tool created by the HDF Group. I used this tool to go in to the data files exactly as they were downloaded from NASA and look at the data. The screen shot to the left shows the values of Channel 4 QA that I reported. Any value other than zero means the channel failed QA. As you can see, all of Channel 4's readings failed. Channel 4 is labeled as "3" because the numbering in the viewer starts at zero rather than one.

Checking Text Data
HDFView is an HDF viewer, not an HDF-EOS viewer. And the Aqua AMSU Level 1B files are HDF-EOS files. This means HDFView can show you the HDF parts of the file, but not the EOS parts of the file. To look at the EOS data in the file, I converted it to text and opened it up in a text editor. Conversion to text was done by taking the original NASA data file and running it through ncdump with no arguments other than the file name to be converted.

The two screen shots below show samples of EOS data for two different files. You can see the file names as the titles to the window. Looking at the highlighted data, you can see the name of the data shown and its value. The data is NumBadData and the value for both files is 1350.

These types of checks were done on about 20 randomly selected files to ensure the errors being reported by the QA code were actually due to the data in the files, not due to bugs in the QA code itself.

Previous Articles In This Series:
NASA, UAH Notified Of QA Spot Check Findings
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

References:
UAH January Raw Data Spot Check
Looking At The Aqua Satellite Data

NASA, UAH Notified Of QA Spot Check Findings

2010-02-25T07:05:00.006-05:00

I've sent off the results of my QA Spot Check to the Aqua team at NASA and the Dr. Christy and Dr. Spencer at UAH. I couldn't find anything wrong with my results, so it's time to let the professionals have their input.

Here's a copy of the text that was sent:

I'm writing with questions regarding the January, 2010 Aqua satellite AMSU Level 1B data. I checked various QA flags in the data and found the following results:

● Of the 7410 files containing January data, 7386 of them had their automaticQualityFlag marked "suspect" and another 24 of them had the flag marked "failed". None passed QA.
● Of the 7386 suspect files, all of them had a "Good Data" percentage of 93.33334 percent. Here, "Good Data" is taken as the result of subtracting bad data, special data, and missing data from total data. "Good Data" is then divided by total data to get the percentage of "Good Data".
● Of the 333,450 Channel 4 readings for January none of them passed QA. All of them in the files marked "suspect" had been marked as failing QA and, obviously, the ones in the files marked "failed" were in files that failed QA and should not be used.

My questions are:

● Is it considered normal to have zero Level 1B AMSU data files for a month pass QA?
● Is it normal for all Level 1B AMSU data files for a month to have the exact same numbers for bad data, missing data, special data, and total data?
● Doesn't the statistics engine used for AMSU limb adjustment require valid data from channel 4 in order to correctly adjust channel 5 data?

Additionally, I asked Dr. Christy and Dr. Spencer if channel 5 from Aqua's AMSU is used to produce UAH anomalies. In an article on WUWT, Dr. Spencer said it is, but I just wanted to double check to make sure I understood correctly.

Previous Articles In This Series:
About The Aqua Satellite Project
UAH January Raw Data Spot Check
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

References:
How the UAH Global Temperatures Are Produced

About The Aqua Satellite Project

2010-02-24T12:57:00.003-05:00

Folks are downloading the code from the Aqua Satellite Project, so I wanted to do a short post about the project.

Purpose
The purpose of the project is to provide a set of tools for accessing and using data from the Aqua satellite. The goal is to be able to replicate as much of the processing done to the data as allowed by law.

State
The project is in its very early stages. More will be added over time, and if you find any mistakes in what's there, please let me know and I'll correct it.

Aqua Satellite Book

I've been working on a book about the Aqua satellite as part of the project. This book isn't a coffee table book or an "executive summary". It digs into the details of how the Aqua satellite works. Because people are downloading the Aqua satellite code, I've added a "early access" version of the book to the download section of the blog under the name Aqua Satellite (PDF). So far, the book is over 50 pages long and there's more to come. Use this book to come up to speed on the technical aspects of the satellite and its related data and processing code.

Short Source Code Overview
I'll close out by giving a brief overview of the code I used to do the QA spot check from this post. Please remember that this is early beta code.

HDFReader.h and HDFReader.cpp - These files contain the code to read a generic HDF-EOS file. The code works, but requires an ASCII version of the HDF-EOS file rather than the binary files that come from NASA. Use the utility ncdump with only the filename as an argument to convert NASA's binary files to text. The Aqua Satellite book has instructions on how to get ncdump.

AMSUReader.h and AMSUReader.cpp - These files are specializations of HDFReader designed to make it easier to read AMSU HDF-EOS files.

CommandLineArgs.h and CommandLineArgs.cpp - Defines a C++ class for working with command line arguments passed to a program.

FileUtils.h and FileUtils.cpp - Provides some cross-platform file utilities.

main.cpp - The entry point to the QA spot check code I wrote.

AMSUQA.h and AMSUQA.cpp This file contains the actual QA spot check code. It's translated to pseudo-code below.

   Load AMSU Data

   // Check QA
   Set NumberOfGoodData Equal To NumberOfData - NumberOfBadData - NumberOfMissingData - NumberOfSpecialData
   Set PercentGoodData Equal To NumberOfGoodData Divided By NumberOfData
   Multiply PercentGoodData By 100

   Foreach scanline
      Foreach channel
         If channel Failed NASA QA, Add 1 To Channel's Bad Count
      End Foreach
   End Foreach

   // Write QA results
   Open Output File Using Value Of AutomaticQualityFlag As Filename
   Write Out Name Of AMSU Input File
   Write Out Percent Good Data
   Foreach Channel
      Write Out Channel's Bad Count
   End Foreach

   Close Output File

References:
UAH January Raw Data Spot Check

UAH January Raw Data Spot Check

2010-02-23T00:10:00.017-05:00

Update 2:
The C++ source code used to run these tests has been added to the Downloads section of the blog under the name Aqua Satellite Project (BETA).

Update:
While I said words to this effect at the end of the post, I wanted to make sure it's clear: These findings are preliminary. Although nearly two months of research has gone into them more needs to be done.

In the meantime, folks wanting to see the data for themselves can download it from NASA by following the steps here: UAH Satellite Data and downloading the viewers by following the steps here: Looking At The Aqua Satellite Data.
======

Killing two birds with one stone, I decided to give my AMSU C++ code a bit of a workout by doing a Quality Control spot check on the January UAH raw data. I downloaded all the January Level 1B data for the AMSU from NASA. There are 7410 files for the month of January. Here's what I found:

No Files Passed NASA's QA. None.
Of these 7410 files, 7386 of them had their automaticQualityFlag marked "suspect" and another 24 of them had the flag marked "failed". Add that up and it's all 7410 files for January were either suspect or failed. No a single file was marked as "passed".

Of the 7386 suspect files, all of them had a "Good Data" percentage of 93.33334 percent, meaning that's the percentage of data in each file that's considered reliable. There's no way such a number would actually be identical out to one one hundred thousandths of a percentage 7386 times out of 7386 tries, so I figured my code that was reading the wrong data for the good/bad/missing/special values.

Nope, my code was good and manually checking a random selection of about 20 files by opening them up in an editor revealed they all had the identical numbers in them for these fields. In other words, the good/bad/missing/special data flags seem to have hard-coded values, at least for "suspect" data.

Channel 4 Never Works (Which Means Channel 5 Never Works Either)
Each of the the files contains 45 readings from each channel. Of these 45 readings, channel 4 failed 45 times every time. The only exception was in the files that had been marked as "failed". So there was never a good reading from channel 4 for the entire month of January.

This is significant because the statistics engine used to correct the satellite readings for channel 5, the channel UAH uses to get it's anomaly information, requires data from channels 4 and 6.

So this means they're either using a non-published algorithm to calculate the anomalies, or the anomalies are wrong.

Screen Shoot Of QA Spot Check Results

(Click for larger image)

Disclaimer
Here are a few things I think of that may affect the validity of this spot check:

I'm not a scientist, I'm a software engineer.
Prior to working on the Aqua satellite as documented in this blog, I had no experience on working with satellite data.
Despite having checked my code, it is beta code, and I may have missed errors.
It's possible that data files are not given "passed" status (as opposed to "suspect" and "failed") until some QA procedure that takes place after Level 1B occurs.
It's possible the order of channel information in the data file does not represent the actual channel number. In other words, the fourth channel in the data file may not be channel 4.
UAH uses other satellites in addition to the Aqua AMSU.

While I think it's important to point out these possible concerns, I also think it's important to point out I don't see any of them as serious threats to reversing what I'm seeing in the data now.

Still, further research needs to be done to triple check these things, and that's what I'm working on now. If I don't come across anything that changes my views, I'll forward these results to Dr. Spencer and Dr. Christy for their input.

Conclusion
At this point, while admitting the possibility of an error on my part, I'd have to say it seems to me the January AMSU data has no validity at all. Nor does anything derived from it. The hard-coded quality values and the complete failure of channel 4 both look like show-stoppers to me.

Previous Posts In this Series
So, About That January UAH Anomaly
A Note On UAH's High January Temperature

References:
The Limb Adjustment of AMSU-A Observations: Methodology and Validation

Calculating Integrals In The Climate Scientist Starter Kit

2010-02-21T13:42:00.000-05:00

Since I showed you how to calculate the derivatives of CO2, I figured I'd show you how to calculate the integral as well. Just like calculating derivatives, calculating integrals is surprisingly easy.

Calculating Integrals In The Climate Scientist Starter Kit
The first thing we need to know is just what an integral is. Another name for integral is "area under the curve", and that's just what an integral is, the area from the bottom of the graph to the place where the lines are. To demonstrate this visually, I plotted a graph that has CO2 twice, once as a line and once as a set of bars. The distance between the bars was set to zero. Here's what the graph looks like:

You can see that when CO2 is plotted as a set of bars, it completely fills up the area under the curve of CO2 plotted as a line. This is exactly what an integral is! So we know there's enough information just from the CO2 values to get the integral. So how do we get the actual number value for the integral?

It's simple. We add up the values for all the bars. This can be done with a simple sum() function:

=SUM(COLUMN_NAME)

Done!

Calculating Integral Ranges
Suppose we want to find the integral of just part of the graph, say between the 5th and 10th months listed. This can be done by taking the integral for each range and subtracting the results.

=SUM(COLUMN_NAME_ROW_1:COLUMN_NAME_ROW_OF_FIRST_VALUE)
=SUM(COLUMN_NAME_ROW_1:COLUMN_NAME_ROW_OF_SECOND_VALUE)
=SECOND_INTEGRAL - FIRST_INTEGRAL

Example:
=SUM(C1:C5)
=SUM(C1:C10)
=SECOND_INTEGRAL - FIRST_INTEGRAL

The first function gives us the integral of columns 1 through 5. The second function gives us the integral of columns 1 through 10. Subtracting the first integral from the second integral gives us the integral of columns 5 through 10.

Previous Posts In This Series:
CO2 Derivatives (Not Al Gore's Kind Of Derivatives)

CO2 Derivatives (Not Al Gore's Kind Of Derivatives)

2010-02-19T18:11:00.001-05:00

A few days ago an article appeared on What's Up With That that claimed to have disproven AGW using statistics. That claim seems rather bold to me, but there were aspects of the paper I found interesting.

The paper talked about the "1st differences" and "2nd differences" of CO2. By this, the author meant the 1st and 2nd derivatives of CO2. Here I mean derivative as the term is used in Calculus, not the CO2 financial derivatives Al Gore wants to sell you on his carbon stock exchange.

This post shows how to calculate the 1st and 2nd derivatives of a variable in the Climate Scientist Starter Kit.

Calculating Derivatives In The Climate Scientist Starter Kit
It turns out it's pretty easy to calculate 1st and 2nd derivatives in the Climate Scientist Starter Kit. A derivative is calculated as dy/dx. In English, this means the change in y divided by the change in x. In this case, the term "y" refers to the y axis and "x" refers to the x axis. So you can calculate a derivative with the following spreadsheet formula:

=(Y_Cell - Previous_Y_Cell) / (X_Cell - Previous_X_Cell)

But its actually even easier than that. Because the X axis represents a series of months, the difference between any two adjoining X values is always 1. This simplifies the formula to:

=(Y_Cell - Previous_Y_Cell) / 1

And because the result of any number divided by one is that number, the formula simplifies even further:

=(Y_Cell - Previous_Y_Cell)

And there's our derivative!

Calculating the 2nd derivative is just the process of getting the derivative of the first derivative. In other words, it's just the same (Y_CELL - Previous_Y_Cell) run against the 1st derivative rather than the original data series.

The 1st And 2nd Derivative Of CO2
The graphs below show the CO2 data from the Climate Scientist Starter Kit, and the 1st and 2nd derivatives of that data.

CO2 Data

CO2 1st Derivative

CO2 2nd Derivative

The 1st derivative tells us the rate of change in the amount of CO2. The 2nd derivative tells us the rate of change in the rate of change in the amount of CO2.

Conclusion
Originally, I had planned to show how the 2nd derivative of CO2 has a good match with changes in cosmic rays. To do this you just lay the normalized cosmic rays on the graph with the 2nd derivative of CO2.

I had done this very quickly with a couple of decades of data when I first read the article I mentioned above. The match was very good. The 2nd derivative of CO2 and cosmic rays changed in lockstep with one another. Unfortunately, when I extended the analysis to the full range of data for the purposes of writing this post, the new range didn't have that nice correlation.

So I have no cool correlation to show you, but now you know how to calculate 1st and 2nd derivatives of data in the Climate Scientist Starter Kit. Well, ok, I do have one correlation to show you. It's between the 1st derivative of CO2 and the Solar Ephemeris. A similar correlation also exists with the 2nd derivative of CO2.

References:
New paper on mathematical analysis of GHG
Climate Scientist Starter Kit v1.5

NSIDC L2_Land Source Code Walkthrough

2010-02-19T03:11:00.009-05:00

In this post we're going to take a closer look at the NSIDC source code that produces their Level 2B Land product. For a very brief overview of the Level 2B Land and other NSDIC source see this post.

Inputs
The source code uses the Level 2A Brightness Temperature file as well as several data files as input. The data files are stored in the anc folder that comes with the source.

NOTE: The data files in the anc folder are binary files, so if you want to see what data they contain, you'll need to write a translation program to convert them to ASCII text. See the the Fortran file ancdata.blk in the src/land/level2/common folder for a description of the data in each data file.

Source Code
The source code is located in the src/land/level2 folder. There are three main groups of source code: C code, Fortran code and common code located in the common folder. The C code is used mainly for QA and I/O purposes. It's the Fortran code that does the real work. The code in the common folder is also Fortran code, stored as reusable blocks.

NOTE: Recall from this post that the AMSR-E scans detects both the H and V polarization of light. The H and V values are what allows the source code to detect land, rain, wind, and other characteristics. It's the source code that actually detects these characteristics, not the satellite.

NOTE: You'll see the term TB used throughout the code. This is shorthand for Brightness Temperature.

The main file is amsre_l2main.f. This file will call all the other code as needed. Overall, the program performs the following processes:

Ingest ancillary databases and external parameters.
Ingest AMSR-E Level 2A TB data.
Grid TB data on EASE-grid projection.
Export gridded TB data.
Perform geophysical retrieval.
Export results/flags as Level 2B land product.

Other source files of note are:

dobsonv3.f Compute the dielectric constant of soil as a function of frequency, soil moisture, sand/clay fractions, and surface temperature.
fmod_pr3ch.f Compute TB as a function of the 6.9, 10.7, and 18.7 frequencies.
fmod_regrs.f Compute vegetation water content and soil moisture as a function of the 10.7, and 18.7 frequencies.
fmod_tb4ch.f Compute R squared and Chi values based on the 10.7, and 18.7 frequencies and values calculated in dobsonv3.f.
fmod_tb6ch.f Compute R squared and Chi values based on the 6.9, 10.7, and 18.7 frequencies and values calculated in dobsonv3.f.

Execution

AMSRE_L2MAIN() executes with 5 command-line arguments:

amsre_l2main.exe INPUT_L2A ANCIL_DIR L2B_DIR GTB_DIR PMC_VER

INPUT_L2A: Filename of the input Level 2A data file

ANCIL_DIR: Directory of ancillary databases

L2B_DIR: Directory of output Level 2B data files

GTB_DIR: Directory of gridded TB data files

PMC_VER: Product maturity code and PGE version number

NOTE: This program is designed to execute on big endian machines. If you're using a little endian machine (Intel machines are little endian) you'll need to add code that converts binary data to little endian values. You'll want to change the file l2ard.c and any code that loads ancillary data larger than than one byte. C++ code for swapping bytes is provided here:

inline bool is_bigendian()
{
const int i = 1;
bool bReturn = ( (*(char*)&i) == 0 );
return bReturn;
} // is_bigendian

inline char* reverse_16_bit (char* in16Bit)
{
char aReversed16Bit[2];

aReversed16Bit[0] = in16Bit[1];
aReversed16Bit[1] = in16Bit[0];
in16Bit[0] = aReversed16Bit[0];
in16Bit[1] = aReversed16Bit[1];

return in16Bit;
} // reverse_16_bit

inline char* reverse_16_bit_if_little_endian (char* in16Bit)
{
if (false == is_bigendian())
return reverse_16_bit(in16Bit);
return in16Bit;
} // reverse_16_bit_if_little_endian

inline char* reverse_32_bit(char* in32Bit)
{
char aReversed32Bit[4];

aReversed32Bit[0] = in32Bit[3];
aReversed32Bit[1] = in32Bit[2];
aReversed32Bit[2] = in32Bit[1];
aReversed32Bit[3] = in32Bit[0];
in32Bit[0] = aReversed32Bit[0];
in32Bit[1] = aReversed32Bit[1];
in32Bit[2] = aReversed32Bit[2];
in32Bit[3] = aReversed32Bit[3];

return in32Bit;
} // reverse_32_bit

inline char* reverse_32_bit_if_little_endian (char* in32Bit)
{
if (false == is_bigendian())
return reverse_32_bit(in32Bit);
return in32Bit;
} // reverse_32_bit_if_little_endian

inline char* reverse_64_bit(char* in64Bit)
{
char aReversed64Bit[8];

aReversed64Bit[0] = in64Bit[7];
aReversed64Bit[1] = in64Bit[6];
aReversed64Bit[2] = in64Bit[5];
aReversed64Bit[3] = in64Bit[4];
aReversed16Bit[4] = in64Bit[3];
aReversed64Bit[5] = in64Bit[2];
aReversed64Bit[6] = in64Bit[1];
aReversed64Bit[7] = in64Bit[0];
in64Bit[0] = aReversed64Bit[0];
in64Bit[1] = aReversed64Bit[1];
in64Bit[2] = aReversed64Bit[2];
in64Bit[3] = aReversed64Bit[3];
in64Bit[4] = aReversed64Bit[4];
in64Bit[5] = aReversed64Bit[5];
in64Bit[6] = aReversed64Bit[6];
in64Bit[7] = aReversed64Bit[7];

return in64Bit;
} // reverse_64_bit

inline char* reverse_64_bit_if_little_endian (char* in64Bit)
{
if (false == is_bigendian())
return reverse_32_bit(in64Bit);
return in64Bit;
} // reverse_64_bit_if_little_endian


template<class NUMBER_TYPE>
inline NUMBER_TYPE reverse_bytes(NUMBER_TYPE inValue)
{
if (2 == sizeof(inValue))
return *(reverse_16_bit((char*) &inValue));
if (4 == sizeof(inValue))
return *(reverse_32_bit((char*) &inValue));
if (8 == sizeof(inValue))
return *(reverse_64_bit((char*) &inValue));
return inValue;
} // reverse_bytes

template<class NUMBER_TYPE>
inline NUMBER_TYPE reverse_bytes_if_little_endian(NUMBER_TYPE inValue)
{
if (false == is_bigendian())
return reverse_bytes(inValue);
return inValue;
} // reverse_bytes_if_little_endian

Output

The Level 2B land product contains surface type, surface moisture, and vegetation water content information for the Earth.

Additional Information

Useful documentation on the algorithms is available at AMSR-E/Aqua L2B Surface Soil Moisture, Ancillary Parms, & QC EASE-Grids.

References:

First Look At NSIDC Source Code

Taking A Look At Raw AMSR-E Data

AMSR-E/Aqua L2B Surface Soil Moisture, Ancillary Parms, & QC EASE-Grids

Update On My Attempts To Get AIRS and UAH Source Code

2010-02-17T23:24:00.003-05:00

UAH Source Code
I noted in this post that Dr. Christy of UAH had told me he was working with John Bates of NOAA to make UAH source code available. Following a suggestion by lucia from The Blackboard, I wrote to Mr. Bates asking him the status of the project.

Mr. Bates mistook me for a member of Dr. Christy's team, and therefore I don't think I should share his exact quotes with the world on a blog. I will say, however, that he confirmed the existence of the project and indicated that it is in its very early stages with no ETA at this time.

So it may be some time before we see UAH source code, but they are working on it.

AIRS Source Code
Recall from this previous post that AIRS tried to hide behind the International Traffic in Arms Regulations" (ITAR) as an excuse not to provide me their source code. My response to them was that I am a U.S. citizen with no plans to export the code, and therefore not covered by ITAR.

The AIRS Science Team Member I had been talking to forwarded the e-mail to someone else at NASA, apparently not on AIRS. Here's what they said.

the AIRS Product Generation Executable (PGE) software has not been cleared for public release for a variety of reasons. It is available only to the members of the AIRS Science Team and those directly working for the AIRS Project. Algorithms and validation results are published in the peer reviewed literature. Data products and executable software are available as mentioned below. Please see links below.

May I ask what your need is for the source code? Perhaps there is some other way we can be of assistance?

So it seems I've gotten past the bogus ITAR claim and now they are hand waving with the excuse of "a variety of reasons" why the code couldn't possibly be made available to the public.

So I wrote them back, giving reason why only the source code can do what I need and asking them to specifically list what the "variety of reasons" are.

I intend to use the source code for personal education, mostly of items not covered in the literature, yet found in the data supplied by the AQUA team and other sources.

As an example, the file AIRS.2010.01.01.240.L1B.AMSU_Rad.v5.0.0.0.G10005063822.hdf contains scan data on channel 5 that exceeds the limits of expected values defined in "The Limb Adjustment of AMSU-A Observations: Methodology and Validation". I'd like to know how the software handles such unexpected values and cannot learn that from the literature. The literature never expected such values and therefore does not cover how to handle them. Similarly, I cannot tell by running pre-compiled executables what happens when such unexpected values are encountered. Only the source code can tell me.

This is one example. There are others, such as experimenting with replacing the statistical error correction algorithms with algorithms that use deterministic data from other sources, such as cosmic ray monitors from Haleakala, Climax, Hermanus, South Africa, McMurdo, Inuvik, and other stations. I cannot get reliable results that can be meaningfully compared to existing products by attempting to reverse engineer the source code from the literature, nor can I use pre-compiled binaries for this purpose.

So I hope that you can help me in this area. As I told Mr. _______, I am a U.S. citizen and have no intention of exporting this code.

Can I ask you, sir, to list in detail what you mean when you say a "variety of reasons" exist that prevent a U.S. taxpayer from seeing computer code to process weather information that was created by funding from U.S. taxpayers? What exactly are these reasons?

Thank you for your time and help with this.

We'll see what they say.

Climate Scientist Starter Kit v2.0 Coming, Part III: Ozone And Pressure

2010-02-16T22:58:00.001-05:00

In this post we look at two new data sets being added to version 2.0 of the Climate Scientist Starter Kit: Ozone and Tropospheric Pressure.

Ozone
Ozone data from the ISCCP project is being added in version 2.0. Because Ozone is an ISCCP product, it'll have regionalized versions of the data as well as a global version.

Tropospheric Air Pressure
Tropospheric Air Pressure data from the ISCCP project is being added in version 2.0. Because this is an ISCCP product, it'll have regionalized versions of the data as well as a global version.

Normalized Data Included
Normalized and reversed normalized data is included for both Ozone and Tropospheric Air Pressure. An example chart showing Normalized South Pole Air Pressure and reversed Normalized South Pole Ozone is shown below. Both datasets have a 13 month running mean added.

Previous Posts In This Series:
Climate Scientist Starter Kit v2.0 Coming
Climate Scientist Starter Kit v2.0 Coming, Part II: Regional Data

AMSR-E And AMSU HDF-EOS C++ Readers Are Done

2010-02-15T23:02:00.001-05:00

The C++ code for reading AMSU Level 1B and AMSR-E Level 2A HDF-EOS files is complete. You can find the code here:

HDFReader.h
HDFReader.cpp
AMSUReader.h
AMSUReader.cpp
AMSREReader.h
AMSREReader.cpp

And this brings to an end the series of posts on being able to read UAH and AMSR-E raw satellite data.

Previous Posts In This Series:
Taking A Look At Raw AMSR-E Data
Taking A Look At Raw UAH Data
HDF Reader C++ Code Is Written!
Some Useful Climate Code
Summary Of Aqua Satellite Data, Computer Code, And Broken Equipment
Aqua Satellite Raw AMSR-E Data
Aqua Satellite Raw UAH Data, Part 2
Aqua Satellite Raw UAH Data, Part 1
Satellite Summary
Aqua Satellite Data Processing
A Note On The UAH And RSS Raw Data
How UAH And RSS Temperatures Are Measured
Overview Of The Aqua Satellite
Looking At The Aqua Satellite Data
UAH Satellite Data
Dangit! More Climate Stuff. UAH and RSS Raw Data

So, About That January UAH Anomaly

2010-02-15T04:08:00.004-05:00

In a previous post I discussed that high anomaly reading from UAH's January measurements, pointing out that it's possible UAH's statistical adjustments to scan readings could have been thrown off by the unusual weather. In another post I discussed how the AMSU data used by UAH has known errors in nearly all of its scans. The error for channel 5, which is used to create the UAH anomaly readings, can go as high as -10 degrees K. In yet another post I discussed the raw scan line data that I've pulled from the AMSU on the Aqua satellite. It just so happens the data I pulled occurred during January.

So in this post, I want to combine those three posts and show why I'm still not convinced the January anomaly reading from UAH was correct. Basically, the data I pulled from January had an error of -14 degrees K in the channel 5 reading, which is 40% higher than the the maximum error according to the journals.

I can't help but wonder how well the statistical methods used to correct the known channel errors react to errors so far out of range of expected values. Actually, I can't help but wonder if errors of this size have ever been tested at all.

Which means I can't help but wonder if the January UAH anomaly has any validity at all.

It also makes me wonder how much additional data falls outside the expected error range. Because I didn't put any effort at all into finding data 40% beyond that range.

References:
Some Useful Climate Code
A Note On UAH's High January Temperature
Taking A Look At Raw UAH Data

First Look At NSIDC Source Code

2010-02-13T13:23:00.002-05:00

In this post we'll take a look at how to order source code from the National Snow and Ice Data Center and what that source code does.

Ordering The Source Code
The source code is stored in 9 folders that NSIDC calls Delivered Algorithm Packages (DAPs). You must order the source code before you download it. This is done by sending off a request to their User Services department using this form. Just tell them you'd like to download the DAPs and they will give you instructions on how to register and download the DAPS via FTP.

Some of the DAPS also use the PORT Mathematical Subroutine Library.

The Code
The code you'll get is written in Fortran and C. There are also data files required by each DAP. Each DAP comes in its own folder and each folder contains a zip file of the source and data files, as well as some read-me files. After you unzip the files, you'll find the source code in the src folder and the data files in the anc folder.

To finish out this post, we'll take a quick look at each DAP.

**L2_Land**
Input File	Level 2A Brightness Temperature File
Output File	Level 2 Land File
Data Produced	Soil Moisture, Vegetation Water Content, Land Surface Temperature, Surface Type
Uses PORT?	Yes
Platform	Linux

**L2_Rain**
Input File	Level 2A Brightness Temperature File
Output File	Level 2 Rain File
Data Produced	Rain Rate, Rain Type, Rain Status, Surface Type
Uses PORT?	No
Platform	Linux

**L2A**
Input File	Level 1A AMSR-E File
Output File	Level 2A Brightness Temperature File
Data Produced	AMSR-E Scan Information
Uses PORT?	No
Platform	Windows 2000/XP

**L2B_Ocean**
Input File	Level 2A Brightness Temperature File
Output File	Level 2B Ocean File
Data Produced	Very Low Resolution Sea Surface Temperature, Low Resolution Sea Surface Temperature, Low Resolution Wind, Medium Resolution Wind, Medium Resolution Vapor, High Resolution Cloud
Uses PORT?	No
Platform	Linux

**L3 Land**
Input File	30 Days Of Level 2 Land Files
Output File	Level 2B Ocean File
Data Produced	AMSR-E Frequency Scans, Soil Moisture, Vegetation Water Content, Land Surface Temperature
Uses PORT?	No
Platform	Linux

**L3 Ocean**
Input File	Level 2B Land Files (Days Worth, Weeks Worth, Month's Worth)
Output File	Level 2B Ocean File
Data Produced	Low Resolution Sea Surface Temperature, Low Resolution Wind, Medium Resolution Wind, Medium Resolution Vapor, High Resolution Cloud
Uses PORT?	No
Platform	Linux

**L3 Rain**
Input File	Level 2A Brightness Temperature File (Month's Worth) Level 2B Rain Files (Month's Worth)
Output File	Level 3 Rain File
Data Produced	Land Rain, Ocean Rain
Uses PORT?	No
Platform	Linux

**L3 Sea Ice**
Input File	Level 2A Brightness Temperature File (Day's Worth)
Output File	Level 3 Sea Ice File
Data Produced	Frequency Scan Data, Ice Data, Snow Depth Data
Uses PORT?	No
Platform	Linux

**L3 Snow**
Input File	Level 2A Brightness Temperature File (Day's Worth), Level 3 Snow Files (5 Days Worth, Month's Worth)
Output File	Level 3 Snow File (Daily, 5 Days, Monthly)
Data Produced	Snow Data
Uses PORT?	No
Platform	Linux

References:
National Snow And Ice Data Center User Services Contact Form
PORT Mathematical Subroutine Library

Source Code!

2010-02-12T15:37:00.000-05:00

Oh yeah! I have source code from the National Snow And Ice Data Center! URAH!

I've just downloaded it and haven't had the time to go over it yet (there's nine different groups of source). But details will follow.

Raw Data

2010-02-11T13:59:00.001-05:00

Dr. John Christy On UAH Source Code

2010-02-11T08:48:00.002-05:00

Following a suggestion by lucia over at The Blackboard, I wrote to Dr. Christy and Dr. Spencer of UAH asking about the public availability of the source code used to process UAH data. Dr. Christy replied:

We are in a program with NOAA to transfer the code to a certified system that will be mounted on a government site and where almost anyone should be able to run it. We actually tried this several years ago, but our code was so complicated that the transfer was eventually given up after six months.

So UAH source code isn't currently available, but they're in the process of working with a NOAA program to make it available. I followed up asking if there was a general ETA for this availability. He replied:

I talked with John Bates of NOAA two weeks ago and indicated I wanted to be early (I said the "first guinea pig") in the program. He didn't have a firm date on when his IT/programming team would be ready to start the transition, so I don't know.

So that's where we stand. No source code now, but there is a process to make it available in the future.

I'd like to thank Dr. Christy for his help in this, and his quick replies (almost immediate!). He also provided me with the official boundaries used to create the various UAH regions in their temperature data, such as extra-tropic boundaries. I've updated the code that produces the ISCCP regional products to exactly match UAH boundaries. More information on that is here.

Previous Posts In This Series:
The Weather And International Traffic in Arms Regulations: Perfect Together

The Weather And International Traffic in Arms Regulations: Perfect Together

2010-02-10T15:52:00.003-05:00

Behind the scenes I've been trying to get ahold of the source code for processing raw UAH weather data from the AMSR unit aboard the Aqua satellite. I sent off a letter to NASA asking for "the source code for producing the AIRS Level 2 standard retrieval product using AIRS IR and AMSU, without-HSB from Level 1 products. "

Here's the answer I got back:

The AIRS source code is ITAR controlled and is not cleared for public release. All AIRS Products are available from the GES/DISC. We also have direct broadcast software to allow local product generation. More information is available at: http://airs.jpl.nasa.gov/data_products/data_products_toc/"

ITAR stands for "International Traffic in Arms Regulations". It covers exporting military weapons to foreign counties or to foreign individuals. The list of items it covers includes satellites and related equipment. The link in their response, by the way, points to various tools, but no source code for actually processing temperature information.

However, ITAR is related to the export of these items. And as I'm a U.S. citizen with no plans to export the code, it's not clear to me how this covers my situation. I sent a response to NASA saying as much and asking for their explanation of the matter. We'll see what they have to say.

UPDATE:
Here are screen shots of the e-mails, with personal information blacked out.

Climate Scientist Starter Kit v2.0 Coming, Part II: Regional Data

2010-02-10T10:06:00.004-05:00

Update:
The region boundaries described below have been updated to exactly match UAH region boundaries. My thanks to Dr. John Christy of UAH for supplying the boundary information.
======

A new feature in the Climate scientist Starter Kit v2.0 is regional data. For all ISCCP data products, the following regions will be supported:

Global
Northern Hemisphere (north of Equator)
Southern Hemisphere (south of Equator)
Tropics (70˚ from South Pole to 110˚ from South Pole)
Northern Extra Tropics (110˚ from South Pole to 120˚ from South Pole)
Southern Extra Tropics (70˚ from South Pole to 60˚ from South Pole)
North Pole (120˚ from South Pole)
South Pole (60˚ from South Pole)

This provides data similar to the UAH temperature data products and allows for analysis on the local level in addition the global level already supported. The ISCCP data in v1.5 includes Clouds and Water Vapor. In v2.0 we'll see the introduction of additional ISCCP data, which wil be discussed in a future post.

Splitting the ISCCP data into regions was done based on the girding of the data product. For equal area grids, the ISCCP data product covers the Earth using the method shown in the diagram below.

This grid was used to create the regional data products.

Cosmic rays will also have regional coverage. This will be done by adding the readings from additional monitors around the globe.

The graph below shows the v2.0 water vapor values as broken up by region.

Normalized Data Included

Normalized data is also included for the regional data. The graph below shows normalized North Pole Water Vapor along with normalized Solar Ephemeris.

Previous Posts In This Series

Climate Scientist Starter Kit v2.0 Coming

Taking A Look At Raw AMSR-E Data

2010-02-09T12:34:00.002-05:00

In this post we're going to take a look at the data in the Level 2A AMSR-E file. But first, a short review on how the AMSR-E scans data.

The AMSRE-E scans at 6 frequencies: 6.9, 10.7, 18.7, 23.8, 36.5, and 89.0 GHz. There are 5 scanning resolutions. The first four scanning resolutions are considered "low res" and stored in an array of data that is 1994 by 243 positions. The fifth scanning resolution is "high res" and stored in an array that is 1994 by 486 positions.

There are also smoothed and unsmoothed versions of the data. The chart below shows which versions of data for each frequency are stored in the Level 2A file.

Link to original table

AMSR-E Spatial Characteristics of Observations
Reso-lution	Foot print	Mean spatial resolution	Channels
Reso-lution	Foot print	Mean spatial resolution	89.0 GHz	36.5 GHz	23.8 GHz	18.7 GHz	10.7 GHz	6.9 GHz
1	75 km x 43 km	56 km	•	•	•	•	•	• o
2	51 km x 29 km	38 km	•	•	•	•	• o
3	27 km x 16 km	21 km	•	•	• o	o
4	14 km x 8 km	12 km	•	o
5	6 km x 4 km	5.4 km	o

• Includes Level-2A (smoothed) data
o Includes Level 1B (un smoothed) data at original spatial resolution

In addition to all this each scan has two components, a vertical (V) component and a horizontal (H) component. These different components are used to measure electric and magnetic values which in turn are used to make determinations about exactly what is being scanned (land, sea, ice, water vapor, etc.)

All of these variations are stored in the Level 2A AMSR-E file. The table below provides the variable names for all this data.

Level 2A Variable names
Variable Name	Data
6.9V_Res.1_TB_(not-resampled)	6.9 V Res 1
6.9H_Res.1_TB_(not-resampled)	6.9 H Res 1
10.7V_Res.2_TB_(not-resampled)	10.7 V Res 2
10.7H_Res.2_TB_(not-resampled)	10.7 H Res 2
18.7V_Res.3_TB_(not-resampled)	18.7 V Res 3
18.7H_Res.3_TB_(not-resampled)	18.7 H Res 3
23.8V_Approx._Res.3_TB_(not-resampled)	23.8 V Res 3
23.8H_Approx._Res.3_TB_(not-resampled)	23.8 H Res 3
36.5V_Res.4_TB_(not-resampled)	36.5 V Res 3
36.5H_Res.4_TB_(not-resampled)	36.5 H Res 3
6.9V_Res.1_TB	6.9 V Res 1 Smoothed
6.9H_Res.1_TB	6.9 H Res 1 Smoothed
10.7V_Res.1_TB	10.7 V Res 1 Smoothed
10.7H_Res.1_TB	10.7 H Res 1 Smoothed
10.7V_Res.2_TB	10.7 V Res 2 Smoothed
10.7H_Res.2_TB	10.7 H Res 2 Smoothed
18.7V_Res.1_TB	18.7 V Res 1 Smoothed
18.7H_Res.1_TB	18.7 H Res 1 Smoothed
18.7V_Res.2_TB	18.7 V Res 2 Smoothed
18.7H_Res.2_TB	18.7 H Res 2 Smoothed
23.8V_Res.1_TB	23.8 V Res 1 Smoothed
23.8V_Res.1_TB	23.8 H Res 1 Smoothed
23.8V_Res.2_TB	23.8 V Res 2 Smoothed
23.8H_Res.2_TB	23.8 H Res 2 Smoothed
23.8V_Res.3_TB	23.8 V Res 3 Smoothed
23.8H_Res.3_TB	23.8 H Res 3 Smoothed
36.5V_Res.1_TB	36.5 V Res 1 Smoothed
36.5H_Res.1_TB	36.5 H Res 1 Smoothed
36.5V_Res.2_TB	36.5 V Res 2 Smoothed
36.5H_Res.2_TB	36.5 H Res 2 Smoothed
36.5V_Res.3_TB	36.5 V Res 3 Smoothed
36.5H_Res.3_TB	36.5 H Res 3 Smoothed
89.0V_Res.1_TB	89.0 V Res 1 Smoothed
89.0H_Res.1_TB	89.0 H Res 1 Smoothed
89.0V_Res.2_TB	89.0 V Res 2 Smoothed
89.0H_Res.2_TB	89.0 H Res 2 Smoothed
89.0V_Res.3_TB	89.0 V Res 3 Smoothed
89.0H_Res.3_TB	89.0 H Res 3 Smoothed
89.0V_Res.4_TB	89.0 V Res 4 Smoothed
89.0H_Res.4_TB	89.0 H Res 4 Smoothed
89.0V_Res.5A_TB_(not-resampled)	89.0 V Res 5
89.0H_Res.5A_TB_(not-resampled)	89.0 H Res 5

The different frequencies are used to measure different attributes of the Earth, as shown in the table below.

Frequency Measurements
Frequency	Measurement
6.9	Sea Surface Temperature, Soil Moisture, Vegetation
10.7	Sea Surface Temperature, Wind Speed
18.7	Wind Speed, Water Vapor
23.8	Unknown
36.5	Cloud Liquid Water
89.0	Rain Rate

With the above information, you can pull the desired data from the Level 2A file. The data for an actual file for 36.5V_Res.4_TB_(not-resampled), which the cloud liquid water frequency, is shown below. It shows a single scan line.

References And Previous Posts In this Series:
AMSRE_4_NWS.ppt
Taking A Look At Raw UAH Data
HDF Reader C++ Code Is Written!
Some Useful Climate Code
Summary Of Aqua Satellite Data, Computer Code, And Broken Equipment
Aqua Satellite Raw AMSR-E Data
Aqua Satellite Raw UAH Data, Part 2
Aqua Satellite Raw UAH Data, Part 1
Satellite Summary
Aqua Satellite Data Processing
A Note On The UAH And RSS Raw Data
How UAH And RSS Temperatures Are Measured
Overview Of The Aqua Satellite
Looking At The Aqua Satellite Data
UAH Satellite Data
Dangit! More Climate Stuff. UAH and RSS Raw Data

Taking A Look At Raw UAH Data

2010-02-08T10:36:00.009-05:00

Now that we can read Level 1B AMSU files, we can look at the raw data used to build UAH temperature anomalies. First, lets review how the AMSU scans temperatures.

Each level 1B data file contains 45 scan lines. These scan lines are along the path of the satellite as it orbits the Earth. Each scan line has 30 footprints. These footprints are perpendicular to the path of the satellite. Finally, each footprint has 15 channels. These channels measure brightness temperatures at different heights of the atmosphere.

A graph of a single scan line containing all 30 foot prints for channel 5 as it was read from an actual Level 1B data file is shown above.

The footprints scan positions further and further away from the satellite. Footprints 15 and 16 and basically scanning directly below the satellite, whereas footprints 1 and 30 are far away from the satellite. The further away from the satellite a footprints are, the larger the error in its temperature measurements.

In the journals, the error estimate for footprints 1 and 30 for channel 5 is said to be about -10 K. In practice, it seems the actual error can be a bit larger, as the error was as much as -14 K in the file I tested, a difference of about 40%.

Moreover, it seems the errors are different for different temperatures. The graph above shows the readings for footprints 1 through 15 for channel 5 for all 45 scan lines in the file. When the temperatures are higher, the differences between footprint 1 and 15 are greater than when the temperatures are lower.

On the left side of the graph, where overall temperatures are higher, the difference between footprint 1 and footprint 15 is about 14 K. On the right side of the graph, where overall temperatures are lower, the difference between footprint 1 and footprint 15 is about 10 K. So in this example, higher temperatures produce an error amount that is about 40% higher than the cooler temperatures.

Recalling that each Level 1B AMSU data file contains six minutes of data, the chart above is saying that in those six minutes a temperature drop of between 3 K and 7 K was detected.

So that's our first look at raw UAH temperature data. Next, we'll take a look at the raw AMSR-E data.

References And Previous Posts In This Series:
HDF Reader C++ Code Is Written!
Some Useful Climate Code
Summary Of Aqua Satellite Data, Computer Code, And Broken Equipment
Aqua Satellite Raw AMSR-E Data
Aqua Satellite Raw UAH Data, Part 2
Aqua Satellite Raw UAH Data, Part 1
Satellite Summary
Aqua Satellite Data Processing
A Note On The UAH And RSS Raw Data
How UAH And RSS Temperatures Are Measured
Overview Of The Aqua Satellite
Looking At The Aqua Satellite Data
UAH Satellite Data
Dangit! More Climate Stuff. UAH and RSS Raw Data

HDF Reader C++ Code Is Written!

2010-02-08T00:10:00.003-05:00

The code to read HDF files is up. The code should be considered a beta release, not so much because of the quality (the code is actually pretty good, if I do say so myself), but because I reserve the right to modify it in the near future.

After having various problems with the code NASA supplies (missing libraries, pre-built binaries that don't link), I decided to just write the reader myself. This post will take a quick look at the code. A follow up post will show some of the results of using the code to read raw UAH data.

The code has been placed in the public domain and you can get it here:
HDFReader.h
HDFReader.cpp

Previous Posts In This Series

Some Useful Climate Code

Summary Of Aqua Satellite Data, Computer Code, And Broken Equipment

Aqua Satellite Raw AMSR-E Data

Aqua Satellite Raw UAH Data, Part 2

Aqua Satellite Raw UAH Data, Part 1

Satellite Summary

Aqua Satellite Data Processing

A Note On The UAH And RSS Raw Data

How UAH And RSS Temperatures Are Measured

Overview Of The Aqua Satellite

Looking At The Aqua Satellite Data

UAH Satellite Data

Dangit! More Climate Stuff. UAH and RSS Raw Data

Prepping The Data File
The AMSR code reads un-compacted ASCII files, and HDF file are compacted binary. So you'll need to change that before reading them with the code. To do that, just use the ncdump tool. The only argument you supply to ncdump in this case is the filename. Redirect the output to a file and use that file as input to the code. And example command line is shown below:

ncdump AIRS.2010.01.01.240.L1B.AMSU_Rad.v5.0.0.0.G10005063822.hdf > MyFile.txt

If you don't yet have ncdump on your computer, see this post for instructions on downloading and using it.

The Code
The underlying code is fairy complex, so we'll skip discussing it. The code is well commented, so you can look there for specific details. Here we'll just discuss how to use the code to read data.

The code sample below shows how to create the reader, declare memory to hold data you'll get from the reader, and read the raw antenna temperatures.

// Supply location of HDF file.
HDFReader oHDFReader(string("./MyFile.txt"));
// 45 scan lines, 30 footprints, 15 channels.
float aData[45][30][15];       

// Load data.
oHDFReader.getVariableData(string("antenna_temp"), aData);

Yup. That's it. Of course, with this version of the code, you need to know that the name of the antenna temperature data is antenna_temp and that it's an array of floats 45 by 30 by 15. Fear not. You can open the ASCII version of the HDF file and see all the variables listed in the variables section near the beginning of the file. These variables will have dimensions listed as constants and those constants are defined in the dimensions section at the beginning of the file.

There's also some hierarchical data in the HDF file and that data is stored in a node structure in HDFReader. To get the root of the node structure, use the getRootNode() method.

Future versions of the code will build on what's here now. Specifically, I want to add methods to access the variables without forcing the programmer to dig through the data file to see what variables are there. Code to do that should be ready soon.

Also coming soon is a look at some of the raw UAH data that gets extracted from these files.

References:
Some Useful Climate Code

Summary Of Aqua Satellite Data, Computer Code, And Broken Equipment

Aqua Satellite Raw AMSR-E Data

Aqua Satellite Raw UAH Data, Part 2

Aqua Satellite Raw UAH Data, Part 1

Satellite Summary

Aqua Satellite Data Processing

A Note On The UAH And RSS Raw Data

How UAH And RSS Temperatures Are Measured

Overview Of The Aqua Satellite

Looking At The Aqua Satellite Data

UAH Satellite Data

Dangit! More Climate Stuff. UAH and RSS Raw Data

HDFReader.h
HDFReader.cpp

A Fix For Climate Scientist Starter Kit Normalized Sunspot Group Numbers

2010-02-06T09:21:00.006-05:00

There's a bug in the normalized Sunspot Group Numbers for version 1.5 of the Climate Scientist Starter Kit. The numbers weren't normalized correctly. This'll be fixed in version 2.0. In the meantime, you can generate correct normalized values yourself for the regular Sunspot Group Numbers by following these steps:

Create a new empty column on the table that has the regular sunspot group number data.
In the top cell of the new column, enter the formula =(ORIGINAL_COLUMN_ROW - MIN(ORIGINAL_COLUMN)) / (MAX(ORIGINAL_COLUMN) - MIN(ORIGINAL_COLUMN))
The parenthesis in that formula are very import. Don’t forget them or rearrange them.
Copy the formula to the rest of the new column.

ORIGINAL_COLUMN stands for the column letter that contains the data you want to normalize, the sunspot group number data. ORIGINAL_COLUMN_ROW stands for the row number in ORIGINAL_COLUMN that’s the same row number you’re entering this formula into on the new empty column.

An example is =(C2 - MIN(C)) / (MAX(C) - MIN(C)).

My apologies for this error.

More On The Length Of Day Delta (LODD)

2010-02-05T14:03:00.003-05:00

I've been doing more research trying to find what causes the length of Earth's day to change every day. This change is called the Length Of Day Delta (LODD) and it seems it's as mysterious to scientists as it is to me.

Previous Posts In This Series
More On Water Vapor, Length Of Day Delta Correlation
Stunning New Climate Correlation

What Causes The LODD?
That's question I wanted to answer as I googled around the net looking for papers or articles about the LODD. But unfortunately, it seems there are almost as many different options on what causes the LODD as there are papers and articles about the LODD. According to one web site, LODD is caused by Global Warming and will make days get shorter. According to another article on the same web site, LODD is caused by Global Warming and will make days get longer.

NASA says the annual changes in length of day are caused by the atmosphere. But someone else at NASA says its the atmosphere, oceans, and land, which narrows it down to everything on the planet. That first NASA article says decade-scale changes are due to changes in the Earth's core.

So to me, it sounds like science doesn't have a solid answer on what causes the LODD, though it may have several good candidates. I told a reader, Charles, that I would look into calculating changes in the LODD based on changes in water vapor. But I think I'm going to have to forgo that promise on the grounds that if professional scientists studying this full time can't provide a solid answer, original research by a software engineer isn't going to settle things either.

But that's ok. I still have the correlations between LODD and other climate factors. I don't need to know causes to examine correlations.

Example Of Another LODD Correlation

In a previous post I showed solar ephemeris data overlaid with LODD data. That graph is reproduced here. We can see from this graph that the monthly scale changes in LODD have a good reverse correlation to the monthly scale changes in the solar ephemeris.

Solar ephemeris data is based on the Earth orbiting the sun. And in this case I think it's safe to say this orbit is causing, directly or indirectly, the monthly changes in the LODD.

That still leaves the causes of the LODD changes over a decade as unknown, and it still leaves us with no explanation as to why there seems to be two different correlations between LODD and water vapor on the decade scale.

References:

More On Water Vapor, Length Of Day Delta Correlation

Stunning New Climate Correlation

Climate Scientist Starter Kit v2.0 Coming

LODD is caused by Global Warming and will make days get shorter

LODD is caused by Global Warming and will make days get longer
NASA: Annual changes in length of day are caused by the atmosphere
NASA: Annual changes in length of day are caused by the atmosphere, oceans, and land