5 Foundations of Reproducibility and Replicability
The slide deck, titled “Foundations of Reproducibility and Replicability”, serves as a introduction to definitions and technical requirements that underpin the modern Open Science vision. It moves beyond the historical context to define the specific “rules of the game” for conducting verifiable research.
5.1 The Turing Way Community matrix
The core of the slide deck is built around the “Turing Way” matrix (Figure 5.1), which deconstructs the definition of reproducibility and replicability based on changes in data and analysis, similarly to others (Nüst et al. 2018), (Kedron et al. 2021), (National Academies of Sciences and Medicine 2019).
Reproducible: A result is reproducible when the same analysis steps performed on the same dataset consistently produces the same answer.
Replicable: A result is replicable when the same analysis performed on different datasets produces qualitatively similar answers; that is, findings holds when using new data.
Robust: A result is robust when the same dataset is subjected to different analysis workflows to answer the same research question and a qualitatively similar or identical answer is produced. Robustness indicates that the conclusion remains stable and independent of different implementations of an analysis (e.g., R vs a Python package for the same analysis flow).
Generalisable: Using different data and different methods to see if the concept applies broadly. That is, combining replicable and robust findings allow us to form generalisable results.
The deck outlines that achieving reproducibility is not an “all or nothing” state but a spectrum of practices that can be adopted incrementally. It emphasises that while the ideal is to be fully reproducible, even small steps towards transparency can significantly enhance the credibility and impact of research.
A key point in the slides is that “publication is not the scholarship; it is merely an advertisement of the scholarship” (Claerbout and Karrenbach 1992). The real scholarship consists of the full software environment, the code, and the data. The focus is then to emphasise that open and reproducibible research refers to the entire process, not just the final paper. The deck argues that research moves progressively from “unreproducible” to “fully reproducible” as more of these associated researhc resources, beyond the final written paper, are made openly available and linked together.
Reflect on the extent to which these terms (reproduction, replication, robustness, generalisation) are widespread in your own discipline or area of research. Which is more predominant in your discipline? And in your research? Of the journals you publish or flag-ship journals in your discipline, do they request articles that support computational reproducibility, empirical reproducibility, or statistical reproducibility?
5.2 Replication of experiments
Replication that goes beyond checking data and rerunning code is fundamental in experimental disciplines. Replication of experiments involves repeating a study to verify findings, test reliability, or establish generalisability, and can be divided into direct replication (repeating identical steps) and conceptual replication (testing the same hypothesis with different methods), alongside systematic extensions (Gómez et al. 2010; Kedron et al. 2025).
Direct replications basically implement the same experimental procedure as previous research. According to (Kedron et al. 2025), they make limited changes to an initial study to verify the claims of the experimental study by testing for conclusion, construct, and internal validity.
Conceptual replications attempt to replicate findings, including tests of hypotheses, by using different experimental procedures or materials (Schmidt 2009). This, for example, implies changes to the population or location to test validity and generalisability to new populations and contexts (Kedron et al. 2025).
[Ongoing]
5.3 Additional resources and readings
If you’re still hesitant to embrace reproducibility, next talk by Dr. Florian Markowetz, based on his paper (Markowetz 2015), will convince you!
In the Understanding Replications and Reproductions chapter of the FORRT Replication Handbook (Röseler et al. 2025), the authors establish a clear taxonomy for “repetitive research”, distinguishing between reproduction (same data) and replication (new data). They argue that reproduction serves as an essential, cost-effective baseline that should precede any replication attempt. By exploring the “closeness” of studies through both theoretical and methodological lenses, the chapter highlights that a “failed” replication doesn’t always mean the original was wrong – it might simply reveal the “boundary conditions” of a theory. Ultimately, the text frames these practices not as antagonistic acts, but as a collaborative spectrum ranging from simple numerical verification to the broad testing of scientific generalisability.