Wrong on an Astronomical Scale

I recently posted an update regarding our R package revisit, aimed at partially remedying the reproducibility crisis, both in the sense of (a) providing transparency to data analyses and (b) flagging possible statistical errors, including misuse of significance testing.

One person commented to me that it may not be important for the package to include warnings about significance testing. I replied that on the contrary, such problems are by far the most common in all of statistics. Today I found an especially egregious case in point, not only because of the errors themselves but even more so because of the shockingly high mathematical sophistication of the culprits.

This fiasco occurs in the article, “Gravitational Waves and Their Mathematics” in the August 2017 issue of the Notices of the AMS, by mathematics and physics professors Lydia Bieri, David Garfinkle and Nicolás Yunes. In describing the results of a dramatic experiment claimed to show the existence of gravitational wages, the authors state,

…the aLIGO detectors recorded the interference pattern associated with a gravitational wave produced in the merger of two black holes 1.3 billion light years away. The signal was so loud (relative to the level of the noise) that the probability that the recorded event was a gravitational wave was much larger than 5𝜎, meaning that the probability of a false alarm was much smaller than 10^-7.

Of course, in that second sentence, the second half is (or at least reads as) the all-too-common error of interpreting a p-value as the probability that the null hypothesis is correct. But that first half (probability of a gravitational wage was much larger than 5𝜎) is quite an “innovation” in the World of Statistical Errors. Actually, it may be a challenge to incorporate a warning for this kind of error in revisit. 🙂

Keep in mind that the authors of this article were NOT the ones who conducted the experiments, nor were they even in collaboration with the study team. But I have seen such things a number of times in physics, and it is reminiscent of some controversy over the confirmation of the existence of the Higgs Boson; I actually may disagree there, but it again shows that, at the least, physicists should stop treating statistics as not worth the effort needed for useful insight.

In that light, this in-depth analysis by the experiments looks well worth reading.

 

Update on Our ‘revisit’ Package

On May 31, I made a post here about our R package revisit, which is designed to help remedy the reproducibility crisis in science. The intended user audience includes

• reviewers of research manuscripts submitted for publication,
• scientists who wish to confirm the results in a published paper, and explore alternate analyses, and
• members of the original research team itself, while collaborating during the course of the research.

The package is documented mainly in the README file, but we now also have a paper on arXiv.org, which explains the reproducibility crisis in detail, and how our package addresses it. Reed Davis and I, the authors of the software, are joined in the paper by Prof. Laurel Beckett of the UC Davis Medical School, and Dr. Paul Thompson of Sanford Research.