5 Foundations of Reproducibility and Replicability
The slide deck, titled “Foundations of Reproducibility and Replicability”, serves as a introduction to definitions and technical requirements that underpin the modern Open Science vision. It moves beyond the historical context to define the specific “rules of the game” for conducting verifiable research.
The core of the presentation is built around the “Turing Way” matrix (Figure 5.1), which categorises research based on two variables: Data and Analysis.
Reproducible: Using the same data and same analysis to reach the same result.
Replicable: Using new data but the same analysis to see if the findings hold.
Robust: Using the same data but different analytical methods to see if the conclusion remains stable.
Generalisable: Using new data and different methods to see if the concept applies broadly.
The deck outlines that achieving reproducibility is not an “all or nothing” state but a spectrum of practices that can be adopted incrementally. It emphasises that while the ideal is to be fully reproducible, even small steps towards transparency can significantly enhance the credibility and impact of research.
A key point in the slides is that “publication is not the scholarship; it is merely an advertisement of the scholarship” (Claerbout and Karrenbach 1992). The real scholarship consists of the full software environment, the code, and the data. The focus is then to emphasise that open and reproducibible research refers to the entire process, not just the final paper. The presentation argues that research moves from “unreproducible” to “fully reproducible” as more of these associated resources, beyond the final written paper, are made available and linked together.
Reflect on the extent to which these terms (reproduction, replication, robustness, generalisation) are widespread in your own discipline or area of research. Which is more predominant in your discipline? And in your research? Of the journals you publish or flag-ship journals in your discipline, do they request articles that support computational reproducibility, empirical reproducibility, or statistical reproducibility? (hint: check The Turing Way’s definitions page (The Turing Way Community 2025).
5.1 Additional resources and readings
If you’re still hesitant to embrace reproducibility, next talk by Dr. Florian Markowetz, based on his paper (Markowetz 2015), will convince you!
In the Understanding Replications and Reproductions chapter of the FORRT Replication Handbook (Röseler et al. 2025), the authors establish a clear taxonomy for “repetitive research”, distinguishing between reproduction (same data) and replication (new data). They argue that reproduction serves as an essential, cost-effective baseline that should precede any replication attempt. By exploring the “closeness” of studies through both theoretical and methodological lenses, the chapter highlights that a “failed” replication doesn’t always mean the original was wrong – it might simply reveal the “boundary conditions” of a theory. Ultimately, the text frames these practices not as antagonistic acts, but as a collaborative spectrum ranging from simple numerical verification to the broad testing of scientific generalizability.
More information on these definitions and terms accross disciplines can be found in (Barba 2018) and (Plesser 2018).