References

Allaire, JJ. 2023. Quarto: R Interface to ’Quarto’ Markdown Publishing System. https://CRAN.R-project.org/package=quarto.

Alston, Jesse M., and Jessica A. Rick. 2021. “A Beginner’s Guide to Conducting Reproducible Research.” The Bulletin of the Ecological Society of America 102 (2): e01801. https://doi.org/10.1002/bes2.1801.

Baker, M. 2015. “First Results from Psychology’s Largest Reproducibility Test.” In Nature News. https://doi.org/10.1038/nature.2015.17433.

Baker, M. 2017. “Check Your Chemistry.” Nature 548 (7668): 485–88. https://doi.org/10.1038/548485a.

Barba, Lorena A. 2018. “Terminologies for Reproducible Research.” In arXiv Preprint arXiv:1802.03311. https://arxiv.org/abs/1802.03311.

Barker, Michelle, Neil P. Chue Hong, Daniel S. Katz, et al. 2022. “Introducing the FAIR Principles for Research Software.” Scientific Data 9 (1). https://doi.org/10.1038/s41597-022-01710-x.

Berti, Johann, Marin Dacos, Gabriel Gallezot, et al. 2022. Passport for Open Science - a Practical Guide Ofr PhD Students. University of Lille. https://www.ouvrirlascience.fr/passport-for-open-science-a-practical-guide-for-phd-students/.

Button, KS, JPA Ioannidis, C Mokrysz, et al. 2013. “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience.” Nature Reviews Neuroscience 14 (5): 365–76. https://doi.org/10.1038/nrn3475.

Chiarelli, Andrea, Lucia Loffreda, and Rob Johnson. 2021. The Art of Publishing Reproducible Research Outputs: Supporting emerging practices through cultural and technological innovation. Zenodo. https://doi.org/10.5281/zenodo.5521077.

Christensen, Garret, Jeremy Freese, and Edward Miguel. 2019. Transparent and Reproducible Social Science Research: How to Do Open Science. University of California Press.

Claerbout, JF, and M Karrenbach. 1992. “Electronic Documents Give Reproducible Research a New Meaning.” In SEG Technical Program Expanded Abstracts 1992. Society of Exploration Geophysicists. https://doi.org/10.1190/1.1822162.

Colavizza, Giovanni, Lauren Cadwallader, Marcel LaFlamme, et al. 2024. “An Analysis of the Effects of Sharing Research Data, Code, and Preprints on Citations.” PLOS ONE 19 (10): e0311493. https://doi.org/10.1371/journal.pone.0311493.

Community, The FORRT. 2025. Framework for Open and Reproducible Research Training. https://forrt.org/.

Cooper, Natalie, and Pen-Yuah Hsing. 2019. Reproducible Code. British Ecological Society. https://www.britishecologicalsociety.org/publications/guides-to/.

Donoho, David L, and Victoria Stodden. 2015. “Reproducible Research in the Mathematical Sciences.” In Princeton Companion to Applied Mathematics. Princeton University Press. https://www.stodden.net/papers/PCAM_20140620-VCS.pdf.

Feldman, Gilad. 2025. “The Value of Replications Goes Beyond Replicability and Is Associated with the Value of the Research It Replicates: Commentary on Isager Et Al. (2025).” Meta-Psychology 9 (October). https://doi.org/10.15626/mp.2024.4326.

Foundation, The National Science, and The Institute of Education Sciences U. S. Department of Education. 2019. Companion Guidelines on Replication & Reproducibility in Education Research. The National Science Foundation. https://www.nsf.gov/pubs/2019/nsf19022/nsf19022.pdf.

Gómez, Omar S., Natalia Juristo, and Sira Vegas. 2010. “Replications Types in Experimental Disciplines.” Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (New York, NY, USA), ESEM ’10. https://doi.org/10.1145/1852786.1852790.

Granell, Carlos, Frank Olaf Ostermann, Daniel Nüst, et al. 2025. “Longitudinal Assessment of Research in GIScience Domain Shows a Positive Impact of Reproducible Research Practices.” EarthArXiv 1 (1): 1–32. https://doi.org/10.31223/X5RJ3W.

Granell, Carlos, Frank O. Ostermann, Daniel Nüst, et al. 2026. “Improving Reproducibility of GIScience Publications Through Novel Reproducibility Guidelines and Revised Review Procedures.” Journal of Spatial Information Science, no. 32: 67–93. https://doi.org/10.5311/josis.2026.32.591.

Ioannidis, JP. 2005. “Why Most Published Research Findings Are False.” PLOS Medicine 2 (8): e124. https://doi.org/10.1371/journal.pmed.0020124.

Ioannidis, JP, TD Stanley, and H Doucouliagos. 2017. “The Power of Bias in Economics Research.” The Economic Journal 127 (605): F236–65. https://doi.org/10.1111/ecoj.12461.

Jolly, M, AC Fletcher, and PE Bourne. 2012. “Ten Simple Rules to Protect Your Intellectual Property.” PLoS Computacional Biology 8: e1002766. https://doi.org/10.1371/journal.pcbi.1002766.

Karathanasis, Nestoras, Daniel Hwang, Vibol Heng, et al. 2022. “Reproducibility Efforts as a Teaching Tool: A Pilot Study.” PLOS Computational Biology 18 (11): e1010615. https://doi.org/10.1371/journal.pcbi.1010615.

Kedron, Peter, Amy E. Frazier, Andrew B. Trgovac, Trisalyn Nelson, and A. Stewart Fotheringham. 2021. “Reproducibility and Replicability in Geographical Analysis.” Geographical Analysis 53 (1): 135–47. https://doi.org/10.1111/gean.12221.

Kedron, Peter, Joseph Holler, and Sarah Bardin. 2025. “A Survey of Researcher Perceptions of Replication in Geography.” Annals of the American Association of Geographers 115 (1): 184–204. https://doi.org/10.1080/24694452.2024.2415695.

Keshav, S. 2007. “How to Read a Paper.” ACM SIGCOMM Computer Communication Review (New York, NY, USA) 37 (3): 83–84. https://doi.org/10.1145/1273445.1273458.

Langton, Samuel, Stijn Ruiter, and Tim Verlaan. 2023. “Describing the Scale and Composition of Calls for Police Service: A Replication and Extension Using Open Data.” Police Practice and Research 24 (5): 523–38. https://doi.org/10.1080/15614263.2022.2102494.

LeBeau, Brandon, Scott Ellison, and Ariel M. Aloe. 2021. “Reproducible Analyses in Education Research.” Review of Research in Education 45 (1): 195–222. https://doi.org/10.3102/0091732X20985076.

Lowenkamp, Christopher T., Francis T. Cullen, and Travis C. Pratt. 2003. “Replicating Sampson and Groves’s Test of Social Disorganization Theory: Revisiting a Criminological Classic.” Journal of Research in Crime and Delinquency 40 (4): 351–73. https://doi.org/10.1177/0022427803256077.

Markowetz, F. 2015. “Five Selfish Reasons to Work Reproducibly.” Genome Biology 16: 274. https://doi.org/10.1186/s13059-015-0850-7.

McDermott, Matthew B. A., Shirly Wang, Nikki Marinsek, Rajesh Ranganath, Luca Foschini, and Marzyeh Ghassemi. 2021. “Reproducibility in Machine Learning for Health Research: Still a Ways to Go.” Science Translational Medicine 13 (586): eabb1655. https://doi.org/10.1126/scitranslmed.abb1655.

Mesnard, Olivier, and Lorena A. Barba. 2017. “Reproducible and Replicable Computational Fluid Dynamics: It’s Harder Than You Think.” Computing in Science Engineering 19 (4): 44–55. https://doi.org/10.1109/MCSE.2017.3151254.

Moassefi, Mana, Pouria Rouzrokh, Gian Marco Conte, et al. 2023. “Reproducibility of Deep Learning Algorithms Developed for Medical Imaging Analysis: A Systematic Review.” Journal of Digital Imaging 36 (5): 2306–12. https://doi.org/10.1007/s10278-023-00870-5.

Monk, Khadija, and Jacek Koziarski. 2023. “Replicating & Reproducing Policing Research.” Police Practice and Research 24 (5): 519–22. https://doi.org/10.1080/15614263.2023.2221075.

Montgomery, Erwin B. 2024. Reproducibility in Biomedical Research: Epistemological and Statistical Problems and the Future. 2nd ed. Academic Press.

Moreau, D., K. Wiebels, and C Boettiger. 2023. “Containers for Computational Reproducibility.” Nature Reviews Methods Primers 3 (50). https://doi.org/10.1038/s43586-023-00236-9.

Morin, A, J Urban, and P Sliz. 2012. “A Quick Guide to Software Licensing for the Scientist-Programmer.” PLoS Computacional Biology 8 (7): e1002598. https://doi.org/10.1371/journal.pcbi.1002598.

MR, Munafò, Nosek BA, Bishop DVM, et al. 2017. “A Quick Guide to Software Licensing for the Scientist-Programmer.” Nature Human Behaviour 1 (0021). https://doi.org/10.1038/s41562-016-0021.

National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. The National Academies Press. https://doi.org/10.17226/25303.

Niven, Daniel J., T. Jared McCormick, Sharon E. Straus, et al. 2018. “Reproducibility of Clinical Research in Critical Care: A Scoping Review.” BMC Medicine 16 (1). https://doi.org/10.1186/s12916-018-1018-6.

Nivette, Amy, Christof Nägel, and Emily Gilbert. 2023. “Examining the Effects of the Killing of George Floyd by Police in the United States on Attitudes of Black Londoners: A Replication.” Police Practice and Research 24 (5): 575–96. https://doi.org/10.1080/15614263.2023.2185241.

Nosek, Brian A., Tom E. Hardwicke, Hannah Moshontz, et al. 2022. “Replicability, Robustness, and Reproducibility in Psychological Science.” Annual Review of Psychology 73 (1): 719–48. https://doi.org/10.1146/annurev-psych-020821-114157.

Noy, Natasha, and Carole Goble. 2023. “Are We Cobblers without Shoes?” Communications of the ACM 66 (1): 36–38. https://doi.org/10.1145/3528574.

Noy, Natasha, and Aleksandr Noy. 2019. “Let Go of Your Data.” Nature Materials 19 (1): 128–28. https://doi.org/10.1038/s41563-019-0539-5.

Nüst, Daniel, Carl Boettiger, and Ben Marwick. 2018. How to Read a Research Compendium. arXiv. https://doi.org/10.48550/ARXIV.1806.09525.

Nüst, Daniel, and Stephen J Eglen. 2021. “CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility.” F1000Research 10 (March): 253. https://doi.org/10.12688/f1000research.51738.1.

Nüst, Daniel, Carlos Granell, and Frank O Ostermann. 2023. Impact of Reproducible Paper Guidelines on Computational Papers: A Longitudinal Study on the AGILE and GIScience Conference Series. OSF. https://doi.org/10.17605/OSF.IO/XZJCH.

Nüst, Daniel, C Granell, B Hofer, et al. 2018. “Reproducible Research and GIScience: An Evaluation Using AGILE Conference Papers.” PeerJ 6: e5072. https://doi.org/10.7717/peerj.5072.

Nüst, Daniel, Frank Ostermann, Rusne Sileryte, et al. 2020. Reproducible Publications at AGILE Conferences: Guidelines for Authors, Scientific Reviewers, and Reproducibility Reviewers. https://doi.org/10.17605/OSF.IO/CB7Z8.

Nüst, Daniel, Vanessa Sochat, Ben Marwick, et al. 2020. “Ten Simple Rules for Writing Dockerfiles for Reproducible Data Science.” PLOS Computational Biology 16 (11): 1–24. https://doi.org/10.1371/journal.pcbi.1008316.

Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251): aac4716. https://doi.org/10.1126/science.aac4716.

Ostermann, Frank O., Daniel Nüst, Carlos Granell, Barbara Hofer, and Markus Konkol. 2021. “Reproducible Research and GIScience: An Evaluation Using GIScience Conference Papers.” In 11th International Conference on Geographic Information Science (GIScience 2021) - Part II, edited by Krzysztof Janowicz and Judith A. Verstegen, vol. 208. Leibniz International Proceedings in Informatics (LIPIcs). Schloss Dagstuhl – Leibniz-Zentrum für Informatik. https://doi.org/10.4230/LIPIcs.GIScience.2021.II.2.

Peng, RD. 2011. “Reproducible Research in Computational Science.” Science 334 (6060): 1226–27. https://doi.org/10.1126/science.1213847.

Pennington, Charlotte. 2023. A Student’s Guide to Open Science: Using the Replication Crisis to Reform Psychology. Open University Press. https://www.mheducation.co.uk/a-student-s-guide-to-open-science-using-the-replication-crisis-to-reform-psychology-9780335251162-emea-group.

Pimentel, João Felipe, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. 2019. “A Large-Scale Study about Quality and Reproducibility of Jupyter Notebooks.” 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), 507–17. https://doi.org/10.1109/MSR.2019.00077.

Plesser, Hans E. 2018. “Reproducibility Vs. Replicability: A Brief History of a Confused Terminology.” Frontiers in Neuroinformatics Volume 11 - 2017. https://doi.org/10.3389/fninf.2017.00076.

Pownall, Madeleine, Flávio Azevedo, Laura M. König, et al. 2023. “Teaching Open and Reproducible Scholarship: A Critical Review of the Evidence Base for Current Pedagogical Methods and Their Outcomes.” Royal Society Open Science 10 (5). https://doi.org/10.1098/rsos.221255.

Röseler, L., L. Wallrich, H. Hartmann, et al. 2025. Handbook for Reproduction and Replication Studies. Zenodo. https://doi.org/10.5281/zenodo.16990114.

Schmidt, Stefan. 2009. “Shall We Really Do It Again? The Powerful Concept of Replication Is Neglected in the Social Sciences.” Review of General Psychology 13 (2): 90–100. https://doi.org/10.1037/a0015108.

Smith, AM, DS Katz, and KE Niemeyer. 2016. “Software Citation Principles.” PeerJ Computer Science 2: e86. https://doi.org/10.7717/peerj-cs.86.

Stark, PB. 2018. “Before Reproducibility Must Come Preproducibility.” Nature 557 (7706): 613–14. https://doi.org/10.1038/d41586-018-05256-0.

Stodden, Victoria, David H Bailey, Jonathan Borwein, R. J. LeVeque, W. Rider, and W. Stein. 2013. “Setting the Default to Reproducible. Reproducibility in Computational and Experimental Mathematics.” Computational Science Research. SIAM News 46 (5): 4–6. https://stodden.net/icerm_report.pdf.

Stodden, Victoria, Jennifer Seiler, and Zhaokun Ma. 2018. “An Empirical Analysis of Journal Policy Effectiveness for Computational Reproducibility.” Proceedings of the National Academy of Sciences 115 (11): 2584–89. https://doi.org/10.1073/pnas.1708290115.

Stodden, V, and SB Miguez. 2014. “Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research.” Journal of Open Research Software 2 (1): e21. https://doi.org/10.5334/jors.ay.

Stupple, Aaron, David Singerman, and Leo Anthony Celi. 2019. “The Reproducibility Crisis in the Age of Digital Medicine.” Npj Digital Medicine 2 (1). https://doi.org/10.1038/s41746-019-0079-z.

Szabo, Csaba. 2025. Unreliable: Bias, Fraud, and the Reproducibility Crisis in Biomedical Research. Columbia University Press.

The Turing Way Community. 2019. The Turing Way: A Handbook for Reproducible Data Science. Zenodo. https://doi.org/10.5281/zenodo.3233986.

The Turing Way Community. 2025. The Turing Way: A Handbook for Reproducible, Ethical and Collaborative Research. Zenodo. https://doi.org/10.5281/zenodo.3233853.

The Turing Way Community, and Scriberia. 2024. Illustrations from The Turing Way: Shared under CC-BY 4.0 for reuse. Zenodo. https://doi.org/10.5281/zenodo.10556824.

Thomas Perry, Rebecca Morris, and Rosanna Lea. 2022. “A Decade of Replication Study in Education? A Mapping Review (2011–2020).” Educational Research and Evaluation 27 (1-2): 12–34. https://doi.org/10.1080/13803611.2021.2022315.

Vilhuber, Lars. 2020. “Reproducibility and Replicability in Economics.” Harvard Data Science Review 2 (4). https://doi.org/10.1162/99608f92.4f6b9e67.

Wilson, Greg. 2019. Teaching Tech Together: How to Make Lessons That Work and Build a Teaching Community Around It. Taylor & Francis. https://teachtogether.tech/.

Xie, Y, JJ Allaire, and G Grolemund. 2018. R Markdown: The Definitive Guide. CRC Press. https://bookdown.org/yihui/rmarkdown/.