SIO 221C Data Analysis Laboratory


Fall 2013

Popular press readings on data analysis

When the popular press starts writing about scientific data analysis methods, maybe we should take this as a warning. Here are two articles of interest:
  1. The Truth Wears Off: Is there something wrong with the scientific method? by Jonah Lehrer, The New Yorker, 13 December 2010.

    The article discusses a phenomenon called the "Decline Effect": astonishing new results are published (often in high-profile journals with lots of media coverage). The results appear to be highly statistically significant, but in subsequent studies, no one ever replicates them as well. And after numerous additional investigations, the original astonishing results are essentially debunked. Why is this? Likely because the original results were in fact an anomaly. Perhaps 100 tests were carried out, and only the case that was significant at the 99% level was published. In fact it was just an outlier. Bayesian statistics might help us avoid this trap, or a healthy willingness to test and retest.

    Note however that journalist Jonah Lehrer has had some integrity issues of his own. Wikipedia provides quite a thorough accounting. Perhaps that means that he's especially well qualified to discuss scientific integrity, or perhaps it means we should doubt his analysis too.

  2. Trouble at the lab, The Economist, 19 October 2013

    This article discusses shoddy statistics and makes a case for verifying published results from other authors and for publishing negative results (if only to provide full data for meta-studies that amalgamate the results of all published work.) While most of the anguish about science and statistics is directed to biomedical research, the issues undoubtedly matter to us too, and this underscores the urgency of us ensuring that we understand statistics.

While I'm at it, let me draw your attention to Nate Silver's book, The Signal and the Noise, which is a fairly readable popular introduction to Bayesian statistics (e.g. the sort of book I would give my brother for Christmas). You might argue with the details of his assessment of the IPCC, but in other regards, he's fairly persuasive.