Historical notes

Reproducibility in Science

Carlos Granell

GEOTEC, Universitat Jaume I

Apr 22, 2024

What is the scientific method?

Popper’s conjectures and refutations

Science progresses through conjectures and refutations

• Scientists are confronted with some question, and offer an answer (conjecture), which might be right or wrong

• Scientists do their best to refute it, or prove it wrong

• Typically it is refuted, rejected, and replaced by a better one

  • And tested over and over…

• If scientists have not been able to refute a theory over a long period of time, then the theory is corroborated

Source: Why should we trust science? Because it doesn’t trust itself

For Popper, the ideas we can most trust are those that have been the most

tried & tested

Advancing science and knowledge is all about

the scientific method!

If science drivers are

Openness

Transparency

Honesty

Integrity

Reproduction (try & test)

Replication (accumulative evidence over time)

Why are reproduction & replication NOT integral parts of the scientific method and modern scientific publishing?

Is reproducibility a new problem?

Let’s go back to the 17th century…

Christiaan Huygens

Robert Boyle

Boyle’s air pump was one of the first documented disputes over reproducibility and experimental science…

Huygens vs Boyle

Huygens observed a new effect (anomalous suspension) in NL

Boyle could not replicate this effect in his own air pump in UK

Huygens went to UK (1663) to personally help Boyle replicate anomalous suspension of water

Source. Note here that non-replication can be good! Here, a new, valid effect during replication was not observed in the original study. That’s how science progresses!

Newton vs Flamsteed

Isaac Newton

John Flamsteed

Flamsteed’s lunar data

SAO/NASA Astrophysics Data System (ADS)

Newton’s request for raw data

In 1695, Sir Isaac Newton wrote a letter to the British Astronomer Royal John Flamsteed, whose data on lunar positions he was trying to get for more than half a year…

“these and all your communications will be useless to me unless you can propose some practicable way or other of supplying me with observations … I want not your calculations, but your observations only.”

Source: (Noy and Noy 2019)

Kollerstrom, N. & Yallop, B. D. J. Hist. Astron. 26, 237–246 (1995).

Let’s go back to the 19th century…

Armand Fizeau

• Experimental physicist in Paris

• His speciality was refining and confirming other people’s results

• The first “replicability reviewer”?

“This is the soul of science because there is no such thing as a fact that cannot be independently corroborated”. Source: How simple ideas lead to scientific discoveries

Let’s go back to 30 years ago, with the birth of the Web…

“Today, few published results are reproducible in any practical sense. To verify them requires almost as much effort as it took to create them originally. After a time, authors are often unable to reproduce their own results! For these reasons, many people ignore most of the literature. but with standards for Fortran, C, UNIX, Latex, postcript, Xwindows, CD-ROM […], there is no longer any excuse for nonreproducible research. It is time to plunge into this new era.”

Extract from Earth Sounding Analysis, Jon Claerbout, 1992

John Claerbout revised his book Earth Soundings Analysis with a valid complaint in A note to the reader (page i).

Let’s go back to Sep 15, 2022

• Rubén Herzog, Universidad de Valparaíso in Chile, published a 2020 article in Scientific Reports

• Paul Lodder, a grad student at the University of Amsterdam, wanted to expand on Rubén’s model

• Paul reproduced the analysis results from the article before looking into how to expand it…

• …but Paul got different results!

Herzog case: Facts

“I [Paul] shared my findings with Rubén by writing up a literate programming report, which allowed me to present my findings in a reproducible manner by including the code that leads to any of the results discussed”.

Source: Retraction Watch

“Because I [Paul] was able to run his code and mine on the same simulated data, it was very clear that the typo caused the results from the paper. I shared my results with Rubén in a written report and he quickly got back to me. He acknowledged that this was indeed an error and thanked me for trying to reproduce his results. He also invited me as a co-author for the resubmission, by which I am truly honored.”

Source: Retraction Watch

Full story on Retraction Watch:

• A grad student finds a ‘typo’ in a psychedelic study’s script that leads to a retraction
• ‘A display of extreme academic integrity’: A grad student who found a key error praises the original author

Herzog case: Lessons

Both collaborated to find out the typo on the shared code

Original 2020 paper was retracted, but both worked in a new version of the paper published in 2023 with a code availability section!

Academic integrity & honesty through transparent and reproducible computational code

Science wins

Is reproducibility a new problem?

It’s not NEW, but it’s STILL a problem.

Data availability in published papers?

(Alsheikh-Ali et al. 2011) assessed 500 papers

• 149 did not subject to any data availability policy

• 208 did not adhere to data availability instructions

• 143 adhered to minimum requirements

• 47 deposited full primary data (~9%)

Reproducibility test in psychology?

(Baker 2015) only could reproduce 39 out of 100 of the published studies in psychology

Reproducibility survey

(Baker 2016) surveyed 1,500 researchers:

• +70% of researchers have tried and failed to reproduce another scientist’s experiments

• +50% have failed to reproduce their own experiments

Is “science is in crisis” narrative wrong?

The new “science is in crisis” narrative is not only empirically unsupported, but also quite obviously counterproductive. Instead of inspiring younger generations to do more and better science, it might foster in them cynicism and indifference. Instead of inviting greater respect for and investment in research, it risks discrediting the value of evidence and feeding antiscientific agendas.

Source: (Fanelli 2018)

Is “science is in crisis” narrative wrong?

Therefore, contemporary science could be more accurately portrayed as facing “new opportunities and challenges” or even a “revolution”. Efforts to promote transparency and reproducibility would find complete justification in such a narrative of transformation and empowerment, a narrative that is not only more compelling and inspiring than that of a crisis, but also better supported by evidence.

Source: (Fanelli 2018)

Should we trust modern science without proper reproduction & replication?

References

Alsheikh-Ali, Alawi A., Waqas Qureshi, Mouaz H. Al-Mallah, and John P. A. Ioannidis. 2011. “Public Availability of Published Research Data in High-Impact Journals.” PLOS ONE 6 (9): 1–4. https://doi.org/10.1371/journal.pone.0024357.

Baker, M. 2015. “First Results from Psychology’s Largest Reproducibility Test.” Nature News. https://doi.org/10.1038/nature.2015.17433.

———. 2016. “1,500 Scientists Lift the Lid on Reproducibility.” Nature 533 (7604): 452–54. https://doi.org/10.1038/533452a.

Fanelli, D. 2018. “Opinion: Is Science Really Facing a Reproducibility Crisis, and Do We Need It To?” Proceedings of the National Academy of Sciences 115 (11): 2628–31. https://doi.org/10.1073/pnas.1708272114.

Noy, Natasha, and Aleksandr Noy. 2019. “Let Go of Your Data.” Nature Materials 19 (1): 128–28. https://doi.org/10.1038/s41563-019-0539-5.