Re-reading their response, I noticed they didn't actually admit in the body of the e-mail to making up all of channel 4's readings. But that's what they do. You can read about it at one of the links they provided here: AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4
===
I just got a response from NASA about the various questions I raised in this post regarding the AMSU on the Aqua satellite.
Basically, their response is that channel 4 failed sometime in late 2007 and now they invent the readings from channel 4 from whole cloth and feed those invented readings into the calculations.
Despite my post on noise, this invention of an entire channel's readings isn't as far-featched as it first sounds. The recurring patterns that exist in all the footprints for all the channels make this far easier to do than if we were dealing with something like land-based thermometer readings. For example, given a single footprint reading on any channel, you can make a reasonable guess as to what all the other footprint readings will be for that channel. The between-channel values also have a very consistent relationship to each other. An example of that consistent relationship is shown in the graphic at the beginning of this post.
They also said that having no automaticQualityFlags for a month marked as "passed" is not a big deal, and that the values for badData, etc., are not the same in every file. On that last one, I'm not so sure. I dug into the files with a text editor and looked at the badData values myself, rather than just trusting my code. They were all the same. But, for now, I'll take their word on it and assume there's something there that I'm not yet understanding.
Anyway, here's their response:
Thank you for your interest in Aqua AMSU-A data. Before your specific questions are answered, understanding the following background information about Aqua AMSU-A will be helpful. All documents referenced here can be found at our public archive at the Goddard Earth Sciences Data and Information Services Center (GES DISC) at http://disc.sci.gsfc.nasa.gov/AIRS/documentation.
Aqua AMSU-A is a microwave sounder that is very similar to AMSU-A instruments flown on many NOAA satellites as well as on the European MetOp-A satellite. Aqua AMSU-A senses the atmosphere in 15 distinct data channels. One of these channels, Channel 4, failed in late 2007, as described in AMSU-A Channel 4 NeDT Update: 20 December 2007 at the document archive.
Aqua AMSU-A products include a variety of flags to indicate data quality. However, not all data quality flags are particularly useful for indicating actual data quality. In particular, ‘AutomaticQualityFlag’ is not an effective flag to check because any datasets less than 100% complete will be marked “suspect.” You may wish to review the AIRS Version 5 Released Files Description document to find more suitable QA flags. We suggest you look at “NeDT,” “state1” and “state2” as being better data quality indicators.
Since the degradation of AMSU-04, the AIRS Level 2 retrieval code does not utilize AMSU channel 4 brightness temperatures. You may want to review AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4 to understand how we are addressing the loss of Channel 4 within our retrieval software.
With that background, let us address your 3 specific questions:
Is it considered normal to have zero Level 1B AMSU data files for a month pass QA?
It is a mistake to characterize a granule in which AutomaticQualityFlag is set to "Suspect" as not passing QA. In fact, this is a “normal” condition since late 2007, and it merely reminds us that the dataset is not 100% complete.
Is it normal for all Level 1B AMSU data files for a month to have the exact same numbers for bad data, missing data, special data, and total data?
All Level-1B data files for the month of January 2010 do not have identical values for NumBadData, etc. For example, data collected over a spacecraft maneuver will reflect the state of the instrument at that time. This situation is typical, and most data for any given month should be in the current nominal state. The typical month will contain 1-3 short intervals of bad data from spacecraft maneuvers.
Doesn't the statistics engine used for AMSU limb adjustment require valid data from channel 4 in order to correctly adjust channel 5 data, the channel which is used to create temperature anomalies provided to the public?
The AIRS retrieval code (statistics engine) does not incorporate a limb adjustment as you have described above. Where reliable sensor data is available, it is applied directly to the appropriate portion of the atmosphere, taking into account the angle of the observation. Of the 2378 infrared and 15 microwave channels available to the AIRS retrieval algorithm, no particular channel is most important in deriving our products. Instead, the unique combination of all these channels of data allows us to develop a very complete and accurate temperature and water vapor profile throughout the entire atmosphere, and that is why our data products are very important to weather forecasting and climate studies.
Again, the AIRS Project believes that many of your technical questions can be answered reviewing the documentation at: http://disc.sci.gsfc.nasa.gov/AIRS/documentation. If you have further questions, we request that you contact us via our “Ask AIRS” portal at http://airs.jpl.nasa.gov/AskAirs/. You may also want to register as an AIRS Data User at http://airs.jpl.nasa.gov/DataRegistration/data/index.cfm. In that way you will be notified whenever a significant announcement regarding the AIRS Project or the AIRS and AMSU-A instruments is issued.
Thank you for your interest in AIRS data,
Previous Posts In This Series:
Posts
This climate science is fond of synthetic data isn't it?
ReplyDeleteSo you get an 'artists impression' of the data that would have been there if it was actually working. Or the cooking channel approach - 'Here's one we made earlier...'
Seriously though, reminds me of most satellite data I've ever worked with. Probably not quite as 'inventive' as some of the Russian stuff though; but never a dull moment, torture it till it confesses.
Chuck
I think you might be a little unreasonable with your characterizations of the CH4 data being made up.
ReplyDeleteI read their answer to say "the satellite was designed to have redundant sensors the failure of 1 is not a concern".
In fact, sensor failure is not uncommon on satellites so it would be crazy to send one up without some sort of redundancy.
Channel 4 is made up, Raven. There's no "backup channel 4 sensor". The data is extrapolated from the readings on the other channels.
ReplyDeleteFrom the link they provided for AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4
[quote]
Option 1 was to keep the Version 5 retrieval algorithm otherwise identical to what it had been, but to eliminate the use of AMSU-A channel 4 from any of the retrieval steps. Option 2 was to attempt to predict the AMSU-A channel 4 observed brightness temperature, based on the observations in the remaining AMSU-A channels
[snip]
As a result of this comparison, Option 2, using predicted values of AMSU channel 4 brightness temperatures in an otherwise unchanged AIRS/AMSU Version 5 retrieval algorithm, was selected for use in future Version 5 processing of AIRS/AMSU data.
[/quote]
You miss the point they made here:
ReplyDeleteOf the 2378 infrared and 15 microwave channels available to the AIRS retrieval algorithm, no particular channel is most important in deriving our products.
They don't need data from Channel 4 or any other data. Their algorithms allow the use of synthetic data in event of a channel failure.
I suspect this is by design because sensors do fail and the satellite designers would have had to plan for this.
P.S.
ReplyDeleteThe redundancy comes in the form of the information (and I'm using that word in the formal sense of Information theory). There's not as much information in these scans as there first appears to be. Most of it is redundant.
Due to the reoccurring patterns in the readings, you could have a single footprint reading and as long as you knew what footprint and what channel that reading was, you could make a reasonable guess at all the other readings on that channel and all the other channels as well.
Post overlap. :)
ReplyDeleteI think we're saying the same thing. The data they use for channel 4 doesn't come from the channel 4 sensor. It's calculated from the readings of the other channels.
We may be agreeing but quibbling about tone.
ReplyDeleteI think the redundancy was planned.
Creating synthentic data to compensate for a channel failure should not be characterized as a hack to cover their butts.
My apologies if I came across that way. They can't be faulted for having a hardware failure, and in the original I _did_ present reasons why I thought their way of handling it was reasonable.
ReplyDeleteThe only point I wanted to make is the redundacy was likely planned and the algorithms for "making up" missing data were probably developed before launch.
ReplyDeleteRaven --
ReplyDeleteI sense that you object to the tone of MagicJava's statement "Basically, their response is that channel 4 failed sometime in late 2007 and now they invent the readings from channel 4 from whole cloth and feed those invented readings into the calculations."
The statement, though, is factually accurate.
If you look again at the NASA doc on how they decided to handle the missing Channel 4 data you will see that they could
1) generate new algorithms and software that did not need to have a Channel 4 data set, or
2) Use the redundancy of the information in the 15 microwave channels to generate a synthetic Channel 4, allowing them to continue to use the same processing algorithms and software.
They chose to generate a synthesized channel 4 data set. "Synthesized" sounds better than "fake" or "invent the readings", but means the same.
That choice of generating a new dataset for channel 4 has the advantage of tending to concentrate the uncertainties of the effect of the missing channel into the new algorithm that is generating the synthetic channel 4 data. The processing algorithm after that remains the same.
IMO, it was a good choice by NASA.
Charlie A
p.s. My bet is that the algorithms for generating the synthetic channel 4 were NOT developed before launch.
Also, the at-launch noise data used by the algorithm to generate channel 4's data is not available online. I wrote to NASA and they sent me the data. I've just now included in the Aqua Satellite Early Access PDF.
ReplyDeleteThe data is called NeDTi data, and the table is on page 15.