Thursday, April 29, 2010

Three-Valued Logic And Irreproducible Results In Science, Part II

Introduction
In the previous post for this series we discussed how Two-Valued and Three-Valued Logic works. In this post we take a look at how Three-Valued Logic (3VL) interacts with the results of science. We'll be using the properties of 3VL's three truth values, True, False, and Unknown, to do this.

In particular we'll be examining how the truth value Unknown of 3VL spreads through science like a virus. In fact, we'll make the claim that scientific results that have unverifiable claims that cannot be independently reproduced are viruses. Any further work that builds upon them is infected and becomes unverifiable and irreproducible itself.

Scientific Claims Must Be Verified
In science, it's not enough to simply make a claim. Claims must be independently verifiable before they are accepted.

When the procedures of the claim are followed and the results of the claim are reproduced, the claim is said to be True. When the procedures of the claim are followed and the results of the claim are not reproduced, the claim is said to be False. An example of a scientific claim being shown False is the original Cold Fusion hypothesis by Martin Fleischmann and Stanley Pons.

However, in situations where the procedures of the claim cannot be followed, we cannot show the claim to be True or False. It is in these cases that the need for 3VL arises, and the truth of the claim is Unknown.

Technical Aside: Cold fusion research has continued since the days of Fleischmann and Pons. Today it is funded by several governments around the world and many researchers in the field believe that the original claims of Fleischmann and Pons have been vindicated. However, several mainstream scientific organizations still disagree, most notably the U.S. Department of Energy and Science magazine. For more information on the ongoing cold fusion research, see the LENR-CANR website.

Example: Aqua Satellite Channel 4 Virus
It helps to have an example, so we'll be using channel 4 of the AMSU on the Aqua satellite. Channel 4 failed completely around December, 2007. In response to this, NASA created a new algorithm and has used it to synthetically create channel 4 data from October 1st, 2007 onward.

While NASA publishes the algorithm used to create synthetic channel 4 values, that algorithm requires certain data that is not available to anyone outside of NASA. Even the folks at NASA's JPL, who are in charge of the Aqua satellite, have said they don't have access to the data.

Without this data it's impossible to verify if the algorithm for synthesizing channel 4 data is correct, even though the algorithm itself is published. Similarly, we cannot demonstrate that the algorithm fails to correctly synthesis channel 4 data. Therefore, the ability of the algorithm to correctly synthesis data must be classified as Unknown because the statement that the algorithm is accurate cannot independently be shown to be True or False.

How The Virus Spreads
To qualify as a virus, the Unknown values must be capable of spreading to other works. To see how this occurs, let's first take a look at how False research is capable of spreading.

We take the example of research attempting to build upon claims that have been demonstrated False, in this case Cold Fusion. The diagram above shows new research that is correct being combined with the results of Cold Fusion. Because Cold Fusion has been shown to be False, the overall conclusions of the research must be False because they require Cold Fusion.

Logically, this situation is captured by a simple predicate: True AND False = False.

The same situation occurs in 3VL when using Unknown, rather than False, values. Because the creation of synthetic data for Aqua's AMSU Channel 4 cannot be shown to be True or False, it is Unknown.  Any research combined with it, no matter how good it is, produces a final result that is also Unknown.

This too is captured by a simple 3VL predicate: True AND Unknown = Unknown.

A concrete example of the spreading of Unknown results in published research is provided by NASA's claims of increased yield due to synthetic channel 4 data. We'll assume that these claims are True and that yields are in fact increasing. However, even with this assumption, we cannot demonstrate that yields should be increasing. Because it cannot be verified that the synthetic channel 4 data is valid, we cannot verify that the synthetic data causes bad data to pass QA or good data to fail QA. The quality of the data in these increased yields is Unknown. This is because the quality of the synthetic data is Unknown.

This cascading of the Unknown value continues through anything that uses the data from these increased yields. In practice, it turns out that all processes referred to by NASA as "Level 2" or higher that use Aqua AMSU data will be infected by the Unknown values. That is, all such data sets have an Unknown truth value themselves due to their dependence upon the increased yield data. These "Level 2" products include:

● Temperature profile from 3 mbar (45 km) to the surface.
● Water vapor profiles.
● Snow and ice coverage.
● Cloud liquid water.
● Cloud-cleared IR radiances.
● Rain Rate.
● Ozone.
Carbon Dioxide Support Products.

Other Examples Of The Virus Spreading
GHCN Data
A while back Willis Eschenbach made the claim that GHCN data at Darwin station was being manipulated to show a warming where none existed in the raw data. Pro-AGW bloggers jumped on this claiming the adjustments were valid.

The problem is the Australian CSIRO Atmospheric Research Center provided no reason why the adjustments were made, stating only that Darwin is a urban site (which should make adjustments go down, not up).

This is an illustrative example of the problem with GHCN adjustments. Even though GHCN provides its raw data and describes its adjustment procedures, it's adjustments cannot be replicated when reasons for the adjustments aren't given.

For this reason, the validity of GHCN data must be classified as Unknown. This Unknown value spreads to anything using GHCN data. This includes the Intergovernmental Panel on Climate Change 4th Assessment Report.

Deep Impact
Deep Impact was a NASA mission to probe a comet by slamming a probe into the comet Tempel I and analyzing the impact. The Deep Impact team at NASA JPL released a photo of the comet with water photoshopped onto its surface (seen at left), a series of medium resolution images of the event, and a chart of the thermal emission spectra of the debris. The chart is shown below.


However, the chart is made of data that's been modified by NASA and the raw data used to generate the Tempel I spectra has never been released. This makes it impossible to verify that the scanners actually produced the results shown in the chart. The resulting unverifiable claims are therefore Unknown.

Curing The Virus
As far-reaching as the consequences of the examples provided here are, we've covered only a small handful of examples. Many more could be provided. Their flow through related work could be tracked and we'd discover that a significant portion of modern science rests upon unverifiable claims.

I think most people wouldn't consider such science to be science at all, but as a problem that stands in the way of science. Fortunately, it's a problem that's easy to fix.

Simply make the claims verifiable.

By making the raw data and computer code used to generate the claims publicly available, and by noting why changes are made to raw data, claims that are currently unverifiable can be demonstrated to be True or False.

And that is the whole reason science exists in the first place.

References
Three-Valued Logic And Irreproducible Results In Science, Part I
Cold Fusion claims by Martin Fleischmann and Stanley Pons - Wikipedia Entry
NASA Responds To FOIA Request
AIRS/AMSU/HSB Version 5 Modification of Algorithm to Account for Increased NeDT in AMSU Channel 4
AMSU - Wikipedia Entry
AIRS/Aqua Level 2 Carbon Dioxide Support Products
The Smoking Gun At Darwin Zero
Willis Eschenbach caught lying about temperature trends
Updating Australia’s high-quality annual temperature dataset
GHCN V.2 Raw Data
GHCN Quality Control, Homogeneity Testing, and Adjustment Procedures
GHCN-Monthly Version 2 Introduction
Deep Impact
LENR-CANR website

22 comments:

  1. You are completely wrong about cold fusion. It was replicated at high signal to noise ratios in hundreds of major laboratories. I have a collection of 1,200 peer-reviewed journal papers on cold fusion copied from the library at Los Alamos. See:

    http://lenr-canr.org

    I suggest you review some of this literature before commenting on this subject. As things stand, your conclusions are a classic case of garbage in, garbage out.

    ReplyDelete
  2. Mr. Rothwell,

    As I noted, I was referring only to the original research by Fleischmann Pons, which could not be replicated.

    I'm aware that cold fusion research is ongoing and there are researchers who feel they've produced successes in this area, and others who disagree. As a software engineer, it's not my place to try and settle physics disputes like that and I wasn't trying to do so.

    ReplyDelete
  3. You wrote:

    "I was referring only to the original research by Fleischmann Pons, which could not be replicated."

    That is incorrect. The original research by Fleischmann and Pons was replicated, including the excess heat claims, neutrons and later tritium and helium production commensurate with a plasma fusion reaction. For example, there are roughly 130 peer-reviewed papers describing the heat replications. The effect has been observed roughly 17,000 times worldwide, according to a tally published by the Institute of High Energy Physics, Chinese Academy of Sciences.

    The authors and organizations that conducted these studies all say they replicated Fleischmann and Pons, and they all cite the original work. For example, EPRI wrote: "This work confirms the claims of Fleischmann, Pons, and Hawkins of the production of excess heat in deuterium-loaded palladium cathodes at levels too large for chemical transformation." (McKubre, M.C.H., et al., Development of Advanced Concepts for Nuclear Processes in Deuterated Metals, TR-104195. 1994, Electric Power Research Institute.)

    It is rather odd that you dispute the authors. They probably know a great deal more about cold fusion than you do.

    ReplyDelete
  4. Oops. Correction. Worldwide replication runs were estimated at 14,700. See J. He, Front. Phys. China (2007) 1: 96 102, Table 1. Reproducibility is 83%.

    This tally can only be successful runs reported in the literature. I doubt that reproducibility overall is 83%. Although for the most successful techniques at Mitsubishi, Toyota, U. Osaka and the U.S. NRL reproducibility is 100%. The NRL repeated the experiment hundreds of times in a row last year with automated equipment, completely degassing the sample and thus starting over from scratch. It worked every time. See Kidwell et al., NRL, ICCF-15 slides.

    ReplyDelete
  5. Is this the slideshow you're referring to?
    http://newenergytimes.com/v2/library/2010/2010KrivitS-ACS-PRINT.pdf

    It says the presentation was bumped from the review due to a "tight schedule". Why would they bump a presentation that proves cheap, powerful desktop fusion?

    Mr. Rothwell, I'm not trying to paint cold fusion as bunk. LENR is a legitimate area of research. But one that seems to be in its very early stages.

    ReplyDelete
  6. You wrote:

    "Is this the slideshow you're referring to?
    http://newenergytimes.com/v2/library/2010/2010KrivitS-ACS-PRINT.pdf"

    I have not referred to any slides. I referred to the EPRI publication, which you can download from EPRI:

    http://my.epri.com/portal/server.pt?Abstract_id=TR-104195

    Or here:

    http://lenr-canr.org/acrobat/McKubreMCHdevelopmen.pdf

    (This version is smaller and searchable.)


    "It says the presentation was bumped from the review due to a 'tight schedule'. Why would they bump a presentation that proves cheap, powerful desktop fusion?"

    All of the presentations described cheap, powerful desktop fusion. There were 49 presentations at this American Chemical Society meeting. That is more than they could fit in the oral presentation time over 2 days. Several papers -- including this one -- were pushed to poster sessions only. That often happens at conferences.

    The ACS conference schedule is here:

    http://lenr-canr.org/Collections/ACSMarch2010program.pdf


    "LENR is a legitimate area of research. But one that seems to be in its very early stages."

    That is correct. However, your assertion was that the original claims made by Fleischmann and Pons were not replicated. That is incorrect, as you see in the literature.

    ReplyDelete
  7. Mr. Rothwell,

    From the paper you provided at:
    http://lenr-canr.org/acrobat/McKubreMCHdevelopmen.pdf

    [quote]
    Nonetheless, the calorimetric results reported to date make it evident that the experimental hypothesis referred to above cannot be supported as stated; additional criteria must be satisfied.
    [/quote]

    Which is pretty much what I said.

    You can turn a False to True with additional work and perhaps cold fusion claims will one day be shown to be True, but that currently doesn't seem to be the case.

    But even if they should forever remain False, the fact that they're testable puts us far ahead of claims that are Unknown. We learn things even from claims that are ultimately False, but unverifiable claims like the quality of synthetic data on the Aqua AMSU channel 4 never get us anywhere.

    ReplyDelete
  8. You quoted EPRI:

    "[quote]
    Nonetheless, the calorimetric results reported to date make it evident that the experimental hypothesis referred to above cannot be supported as stated; additional criteria must be satisfied.
    [/quote]

    Which is pretty much what I said."

    With all due respect, that is not what you said. You said the "results of the claim are not reproduced." You said nothing about the experimental hypothesis.

    An experiment and a hypothesis are two separate things. It often happens that the first researchers to discover a phenomenon have only an incomplete or incorrect hypothesis to explain it. However, when other researchers do the same experiment and observe the same results, everyone agrees they have observed the same phenomenon. They have still replicated, even though the original hypothesis is abandoned. The hypothesis was modified and extended in the case of cold fusion, not abandoned. As far as I know, no one other than Krivit claims this is anything other than fusion (Ref. the slides you cited).

    There is as yet no complete hypothesis (or theory) to explain the cold fusion results. There are various working rules, like the ones described in the EPRI publication, such as the need for high loading and flux.

    There are, of course, countless other phenomena proved to exist by experiment that cannot be explained by present-day theory.

    ReplyDelete
  9. Mr. Rothwell,

    You make a fair point and I've changed the original text to "hypothesis" to clarify the meaning.

    ReplyDelete
  10. With temperature records what has been recorded is all there is. There can be no replication of the historic record (obviously).

    I too can see no reason why all relevant information is not released. GHCN I believe has released all code and all data?

    With Darwin there were a number of changes unearthed - early records were made not using ANY form of stevenson screen (thermometers were moved to shade as day progressed!) then in 1941 the PO was bombed and the location was moved to darwin airport (a significant change) This plot I made shows this 1941 change (no measurements made and a step change in records:
    http://climateandstuff.blogspot.com/

    There is a similar discontinuity in may june 1994
    GHCN adjustments add up to similar amounts but are adjusted over time.

    Without waiting another 150 years with multiple temperature measurement systems in place how can GW be verified? 150 years if GW is valid will be far too late. Suggest a proof that GW is false (using no proxies, ice cores, historical instrument readings - or suggest an accurate record (is CET ok? Its derivation is recorded))

    ReplyDelete
  11. thefordprefect,

    [quote]
    With temperature records what has been recorded is all there is. There can be no replication of the historic record (obviously).
    [/quote]
    Well, no. There's the steps taken to recreate past temperatures and ongoing adjustments to past recorded temperatures. These steps should be fully disclosed so others can reproduce the results.

    [quote]
    GHCN I believe has released all code and all data?
    [/quote]
    Yes, the have. But without knowing what adjustments were made and why, we're left guessing, not verifying.

    [quote]
    With Darwin there were a number of changes unearthed - early records were made not using ANY form of stevenson screen (thermometers were moved to shade as day progressed!) then in 1941 the PO was bombed and the location was moved to darwin airport (a significant change) This plot I made shows this 1941 change (no measurements made and a step change in records:
    http://climateandstuff.blogspot.com/
    [/quote]
    You have a very nice site, btw (I love movies :) ) . Anyway, as to Darwin, why does moving a thermometer from the post office to the airport mean that the measurements should be adjusted upward?

    I'm not saying there's no valid reason, only that no reason has been given. And without a valid reason, it can't be verified the change itself is valid. As to later adjustments, I don't see any reason for those either. Again, that doesn't mean there's not a reason. It just means no reason was given and the adjustments can't be verified.

    [quote]
    Without waiting another 150 years with multiple temperature measurement systems in place how can GW be verified? 150 years if GW is valid will be far too late.
    [/quote]
    It's up to the scientists themselves to tell us how the theories can be validated. For example, Hansen predicted that the stratosphere would cool because of global warming. The stratosphere hasn't cooling in about 15 years. Does this invalidate global warming?

    ReplyDelete
  12. wiki
    The stratosphere is situated between about 10 km (6 miles) and 50 km (31 miles) altitude above the surface at moderate latitudes, while at the poles it starts at about 8 km (5 miles) altitude.

    Chosing 21km data fromChristies satellite temperature slope is
    -9E-5 per day -0.03C/year
    so not sure where you got your no change for 15 years from!.

    http://img511.imageshack.us/img511/3443/amsutempsinc21km.png

    green dotted trace is strat.

    ReplyDelete
  13. The data comes from RSS:
    http://www.ssmi.com/data/msu/graphics/plots/sc_Rss_compare_TS_channel_tts_v03_2.png

    There is a downward trend from the beginning of the series in 1980, but since 1995 (15 years ago) it's been flat.

    But my intent was not to argue stratosphere temps. I was asking if Dr. Hansen has ever published the conditions that would show global warming to be false.

    ReplyDelete
  14. P.S.

    I grabbed the upper stratosphere image by accident.

    here the lower stratosphere:
    http://www.ssmi.com/data/msu/graphics/plots/sc_Rss_compare_TS_channel_tls_v03_2.png

    ReplyDelete
  15. You wrote:

    "You make a fair point and I've changed the original text to 'hypothesis' to clarify the meaning."

    That's an improvement. Perhaps you should also say that it is your opinion that the hypothesis was false, and that opinion is contrary to nearly every expert and every published paper. The experts agree that the hypothesis was correct but incomplete.

    When you make a statement not supported by the literature, it is good form to say so. Of course you have a right to your opinion, but your readers would probably want to know that thousands of distinguished scientists disagree, and only a handful think you are right.

    ReplyDelete
  16. Mr. Rothwell,

    The original cold fusion clams were rejected like few other scientific claims have ever been rejected. Now you are saying that there are more scientists that can prove cold fusion works than can fit into a symposium. What changed in the middle of these two events?

    Money changed.

    Now there are governments paying scientists to do research in this area and those scientists are all saying "Yes, cold fusion is great." This is pretty much what scientists do whenever money is tossed around.

    I used to work at the Navy's SPAWAR project back in the Star Wars days. We all knew Reagan's Star War's project would never work, but we keep working and we produced results that _did_ work even those results didn't change the fact that the technology central to Star Wars _didn't_ work and never would.

    Now SPAWAR is working on cold fusion and producing results. I see little reason to think they're doing anything different now than what they were doing back in the 80s, which is taking the governments money and reporting whatever success they can.

    Maybe I'm wrong, and I'll certainly grant you that the original cold fusion research was judged too harshly by both the press and the scientific community. But the fact remains that the original research was soundly rejected and the original claims withdrawn.

    Resurrecting those claims and showing them to be True, if it can be done, is going to take time.

    ReplyDelete
  17. You wrote:

    "What changed in the middle of these two events?

    Money changed.

    Now there are governments paying scientists to do research in this area and those scientists are all saying 'Yes, cold fusion is great.' This is pretty much what scientists do whenever money is tossed around."

    Or as Stan Szpak of SPAWAR put it, scientists believe whatever you pay them to believe.

    However, you are wrong. There is practically no funding available for cold fusion in the U.S., and the political opposition is as strong as ever.


    "I used to work at the Navy's SPAWAR project back in the Star Wars days. We all knew Reagan's Star War's project would never work, but we keep working and we produced results that _did_ work even those results didn't change the fact that the technology central to Star Wars _didn't_ work and never would.

    Now SPAWAR is working on cold fusion and producing results."

    Three people at SPAWAR working on cold fusion. Szpak is retired and still comes into the lab. The other two work on weekends and after hours. No funding has been allocated since 1990. At China Lake, when Miles reported positive results, they cut all his funding, threw him out of the lab and reassigned him to a menial job in the stock room. He got the message and resigned. Note that Miles was designated a "Distinguished Fellow of the Institute" at China Lake.

    There is tremendous political opposition to this research. Robert Park told a large crowd of cheering people that he and Zimmerman (a Clinton appointee) would "root out and fire" any Federal researcher who tries to do an experiment, attends a conference, or even talks about cold fusion. I was there in the audience and I know for a fact that they followed through on those threats.


    "I see little reason to think they're doing anything different now than what they were doing back in the 80s, which is taking the governments money and reporting whatever success they can."

    I suggest you review the literature more carefully. You will see very large and obvious differences, such as the fact that with many techniques, reproducibility is now 100%.


    "Maybe I'm wrong, and I'll certainly grant you that the original cold fusion research was judged too harshly by both the press and the scientific community. But the fact remains that the original research was soundly rejected and the original claims withdrawn."

    The research was rejected but the claims were not retracted or withdrawn. Thousands of scientists replicated. Not one of them has withdrawn. When I say the results were rejected, I do not mean only that most journals refused to publish them, or that people criticized them at conferences. I mean that Fleischmann, Pons, Miles, and many others were fired or forced out. Their experiments were trashed with horse manure; their equipment and papers stolen; their houses vandalized, and they were accused in the Washington Post, Time magazine, the New York Times, Sci. Am., New Scientist and elsewhere of being “frauds, lunatics and criminals,” which made it impossible for them to get funding. Schwinger resigned from the APS in protest, and wrote:

    "The pressure for conformity is enormous. I have experienced it in editors’ rejection of submitted papers, based on venomous criticism of anonymous referees. The replacement of impartial reviewing by censorship will be the death of science."

    http://lenr-canr.org/acrobat/SchwingerJcoldfusiona.pdf

    When scientists stand by their results at great personal sacrifice, even at the cost of their careers, you can be sure they are not doing it for money, despite Szpak’s dictum. There are still a few like that. They are old fashioned people. Most are retired, or dead. When the last one dies, I expect cold fusion will be forgotten. You say it will "take time." I am afraid the researchers do not have time.

    ReplyDelete
  18. Mr. Rothwell,

    If you can provide me with a replacement for cold fusion in the post I wrote, I'll rewrite the post without any reference to cold fusion. The replacement must:

    *) Show why science claims need to be validated.
    *) Have been shown to be False.
    *) Recognizable by non-scientists.

    In return, I will consider whatever debt I may have to the SPAWAR folks for getting my career started and me leaving a year earlier than I said I would, to be paid.

    ReplyDelete
  19. You wrote:

    "If you can provide me with a replacement for cold fusion in the post I wrote, I'll rewrite the post without any reference to cold fusion. The replacement must:

    *) Show why science claims need to be validated.
    *) Have been shown to be False.
    *) Recognizable by non-scientists."

    This is not about obligations or what I should do for you. You made an error. Cold fusion does not fit your criteria. It has been shown to be true, not false. Countless claims were shown to be false, but apart from polywater and a few others, they are obscure or forgotten. They are forgotten precisely because they were not replicated. You would have to look through back issues of physics journals to find them.

    We know that cold fusion is real because it was independently replicated in hundreds of laboratories at high signal to noise ratios. That is the only way we ever know that any claim is true; there are not other criteria. If an effect could be widely replicated at a high s/n ratio and yet still be wrong, the scientific method would not work, and civilization would not exist.

    If there is any obligation, I suppose it is one that you owe to yourself: to read about a subject and learn about it before judging it, and to apply objective standards of science to all claims.

    ReplyDelete
  20. Mr. Rothwell,

    I've added a "Technical Aside" on cold fusion that I hope reflects your side of the dispute and provides a link to your web site.

    Until objections from scientists outside the cold fusion community can be met, I think that's the best I can do.

    ReplyDelete
  21. You wrote:

    "Until objections from scientists outside the cold fusion community can be met, I think that's the best I can do."

    Let me know if you find any objections that have not been met. "Skeptics" have published only 5 or 6 peer-reviewed papers as far as I know. Most of these negative papers are available at LENR-CANR.org. As you see from this one, all of their objections were met by 1990:

    http://lenr-canr.org/acrobat/Fleischmanreplytothe.pdf

    Seriously, if you can find a skeptical paper not in our collection, or one that raises issues that have not been met, I would sincerely appreciate it if you would let me know. My address as on the the front of LENR-CANR.org. I strive to make LENR-CANR.org the world's largest collection of anti-cold fusion papers.

    Here is a summary of the skeptical arguments:

    http://pages.csam.montclair.edu/~kowalski/cf/293wikipedia.html

    ReplyDelete
  22. Mr. Rothwell,

    As I mentioned in my first post, it's not for me to settle disputes in physics.

    For now, I would say take heart in the progress the field has made and the knowledge that if the research is sound, in the long run it will win out.

    To paraphrase an old saying, the mills of science grind slowly, but they grind exceedingly small.

    ReplyDelete