References

All materials described below are (should be) freely available online. If you cannot get to them, let me know. Note that a lot of the listed resources are dynamic and ever changing. That means occasionally links might not work, sites go down. If you notice anything that’s not quite right, please let me know.

References

Aden-Buie, Garrick, Barret Schloerke, JJ Allaire, and Alexander Rossell Hayes. 2023. Learnr: Interactive Tutorials for r. https://rstudio.github.io/learnr/.

Allaire, JJ. 2023. Quarto: R Interface to ’Quarto’ Markdown Publishing System. https://CRAN.R-project.org/package=quarto.

Alsheikh-Ali, Alawi A., Waqas Qureshi, Mouaz H. Al-Mallah, and John P. A. Ioannidis. 2011. “Public Availability of Published Research Data in High-Impact Journals.” PLOS ONE 6 (9): 1–4. https://doi.org/10.1371/journal.pone.0024357.

Alston, Jesse M., and Jessica A. Rick. 2021. “A Beginner’s Guide to Conducting Reproducible Research.” The Bulletin of the Ecological Society of America 102 (2): e01801. https://doi.org/10.1002/bes2.1801.

Baker, M. 2015. “First Results from Psychology’s Largest Reproducibility Test.” Nature News. https://doi.org/10.1038/nature.2015.17433.

———. 2016. “1,500 Scientists Lift the Lid on Reproducibility.” Nature 533 (7604): 452–54. https://doi.org/10.1038/533452a.

———. 2017. “Check Your Chemistry.” Nature 548 (7668): 485–88. https://doi.org/10.1038/548485a.

Barba, Lorena A. 2018. “Terminologies for Reproducible Research.” arXiv Preprint arXiv:1802.03311. https://arxiv.org/abs/1802.03311.

———. 2021. “Trustworthy Computational Evidence Through Transparency and Reproducibility.” Computing in Science & Engineering 23 (1): 58–64. https://doi.org/10.1109/MCSE.2020.3048406.

Barker, Michelle, Neil P. Chue Hong, Daniel S. Katz, Anna-Lena Lamprecht, Carlos Martinez-Ortiz, Fotis Psomopoulos, Jennifer Harrow, et al. 2022. “Introducing the FAIR Principles for Research Software.” Scientific Data 9 (1). https://doi.org/10.1038/s41597-022-01710-x.

Berti, Johann, Marin Dacos, Gabriel Gallezot, Madeleine Géroudet, Sabrina Granger, Joanna Janik, Claire Josserand, et al. 2022. “Passport for Open Science - a Practical Guide Ofr PhD Students.” University of Lille. https://www.ouvrirlascience.fr/passport-for-open-science-a-practical-guide-for-phd-students/.

Biggs, John, and Catherine Tang. 2011. Teaching for Quality Learning at University, 4th Edition. Open University Press.

Bion, R, R Chang, and J Goodman. 2018. “How r Helps Airbnb Make the Most of Its Data.” The American Statistician 72 (1): 46–52. https://doi.org/10.1080/00031305.2017.1392362.

Bryan, Jennifer. 2018. “Excuse Me, Do You Have a Moment to Talk about Version Control?” The American Statistician 72 (1): 20–27. https://doi.org/10.1080/00031305.2017.1399928.

Bryan, Jenny, and Jim Hester. 2020. Happy Git and GitHub for the useR. https://happygitwithr.com/.

Button, KS, JPA Ioannidis, C Mokrysz, BA Nosek, J Flint, ESJ Robinson, and MR Munafò. 2013. “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience.” Nature Reviews Neuroscience 14 (5): 365–76. https://doi.org/10.1038/nrn3475.

Callaghan, S. 2014. “Joint Declaration of Data Citation Principles.” https://www.force11.org/datacitationprinciples.

Caprarelli, Graziella, Brian Sedora, Mia Ricci, Shelley Stall, and Matthew Giampoala. 2023. “Notebooks Now! The Future of Reproducible Research.” Earth and Space Science 10 (12). https://doi.org/10.1029/2023ea003458.

Chiarelli, Andrea, Lucia Loffreda, and Rob Johnson. 2021. “The Art of Publishing Reproducible Research Outputs: Supporting emerging practices through cultural and technological innovation.” Zenodo. https://doi.org/10.5281/zenodo.5521077.

Claerbout, JF, and M Karrenbach. 1992. “Electronic Documents Give Reproducible Research a New Meaning.” In SEG Technical Program Expanded Abstracts 1992, 601–4. Society of Exploration Geophysicists. https://doi.org/10.1190/1.1822162.

Cooper, Natalie, and Pen-Yuah Hsing. 2019. “Reproducible Code.” British Ecological Society. https://www.britishecologicalsociety.org/publications/guides-to/.

Delgado-López-Cózar, E, I Ràfols, and E Abadal. 2021. “Letter: A Call for a Radical Change in Research Evaluation in Spain.” Profesional de La Información 30 (3): e300309. https://doi.org/10.3145/epi.2021.may.09.

Dhar, Payal. 2023. “Octopus and ResearchEquals Aim to Break the Publishing Mould.” Nature. Springer Science; Business Media LLC. https://doi.org/10.1038/d41586-023-00861-0.

Donoho, David L, and Victoria Stodden. 2015. “Reproducible Research in the Mathematical Sciences.” In Princeton Companion to Applied Mathematics, 916–25. Princeton University Press. https://www.stodden.net/papers/PCAM_20140620-VCS.pdf.

Donoho, DL, A Maleki, IU Rahman, M Shahram, and V Stodden. 2009. “Reproducible Research in Computational Harmonic Analysis.” Computing in Science & Engineering 11 (1): 8–18. https://doi.org/10.1109/MCSE.2009.15.

Editorial. 2021. “Good Research Begins Long Before Papers Get Written.” Nature 593 (7857): 8–8. https://doi.org/10.1038/d41586-021-01167-9.

Fanelli, D. 2018. “Opinion: Is Science Really Facing a Reproducibility Crisis, and Do We Need It To?” Proceedings of the National Academy of Sciences 115 (11): 2628–31. https://doi.org/10.1073/pnas.1708272114.

Foundation, The National Science, and The Institute of Education Sciences U. S. Department of Education. 2019. Companion Guidelines on Replication & Reproducibility in Education Research. The National Science Foundation. https://www.nsf.gov/pubs/2019/nsf19022/nsf19022.pdf.

Granell, C. 2019. “Directrices Para Artículos Reproducibles.” https://doi.org/10.17605/OSF.IO/MF9BE.

Granell, C, B Hofer, Daniel Nüst, FO Ostermann, and R Sileryte. 2020. “Reproducibilidad En AGILE: Experiencias, Logros y Recomendaciones.” Revista Cartográfica, no. 100: 155–72. https://doi.org/10.35424/rcarto.i100.668.

Granell, C, Daniel Nüst, FO Ostermann, and R Sileryte. 2018. “Reproducible Research Is Like Riding a Bike.” PeerJ Preprints 6: e27216v1. https://doi.org/10.7287/peerj.preprints.27216v1.

Harrison, Kate. 2018. “Data Management.” British Ecological Society. https://www.britishecologicalsociety.org/publications/guides-to/.

Hasselbring, Wilhelm, Leslie Carr, Simon Hettrick, Heather Packer, and Thanassis Tiropanis. 2020. “Open Source Research Software.” Computer 53 (8): 84–88. https://doi.org/10.1109/MC.2020.2998235.

Hohman, Fred, Matthew Conlen, Jeffrey Heer, and Duen Chau. 2020. “Communicating with Interactive Articles.” Distill 5 (9). https://doi.org/10.23915/distill.00028.

Hong, NPC, T Crick, IP Gent, L Kotthoff, and K Takeda. 2015. “Top Tips to Make Your Research Irreproducible.” arXiv Preprint arXiv:1504.00062. https://arxiv.org/abs/1504.00062.

Ioannidis, JP. 2005. “Why Most Published Research Findings Are False.” PLOS Medicine 2 (8): e124. https://doi.org/10.1371/journal.pmed.0020124.

Ioannidis, JP, TD Stanley, and H Doucouliagos. 2017. “The Power of Bias in Economics Research.” The Economic Journal 127 (605): F236–65. https://doi.org/10.1111/ecoj.12461.

Jolly, M, AC Fletcher, and PE Bourne. 2012. “Ten Simple Rules to Protect Your Intellectual Property.” PLoS Computacional Biology 8: e1002766. https://doi.org/10.1371/journal.pcbi.1002766.

Karathanasis, Nestoras, Daniel Hwang, Vibol Heng, Rimal Abhimannyu, Phillip Slogoff-Sevilla, Gina Buchel, Victoria Frisbie, Peiyao Li, Dafni Kryoneriti, and Isidore Rigoutsos. 2022. “Reproducibility Efforts as a Teaching Tool: A Pilot Study.” PLOS Computational Biology 18 (11): e1010615. https://doi.org/10.1371/journal.pcbi.1010615.

Keshav, S. 2007. “How to Read a Paper.” ACM SIGCOMM Computer Communication Review 37 (3): 83–84. https://doi.org/10.1145/1273445.1273458.

Knuth, DE. 1984. “Literate Programming.” The Computer Journal 11 (2): 97–111. https://doi.org/10.1093/comjnl/27.2.97.

LeBeau, Brandon, Scott Ellison, and Ariel M. Aloe. 2021. “Reproducible Analyses in Education Research.” Review of Research in Education 45 (1): 195–222. https://doi.org/10.3102/0091732X20985076.

Leek, JT, and LR Jager. 2017. “Is Most Published Research Really False?” Annual Review of Statistics and Its Application 4: 109–22. https://doi.org/10.1146/annurev-statistics-060116-054104.

Leipzig, Jeremy, Daniel Nüst, Charles Tapley Hoyt, Karthik Ram, and Jane Greenberg. 2021. “The Role of Metadata in Reproducible Computational Research.” Patterns 2 (9): 100322. https://doi.org/10.1016/j.patter.2021.100322.

Macleod, Malcolm, Andrew M. Collings, Chris Graf, Veronique Kiermer, David Mellor, Sowmya Swaminathan, Deborah Sweet, and Valda Vinson. 2021. “The MDAR (Materials Design Analysis Reporting) Framework for Transparent Reporting in the Life Sciences.” Proceedings of the National Academy of Sciences 118 (17): e2103238118. https://doi.org/10.1073/pnas.2103238118.

Markowetz, F. 2015. “Five Selfish Reasons to Work Reproducibly.” Genome Biology 16: 274. https://doi.org/10.1186/s13059-015-0850-7.

McDermott, Matthew B. A., Shirly Wang, Nikki Marinsek, Rajesh Ranganath, Luca Foschini, and Marzyeh Ghassemi. 2021. “Reproducibility in Machine Learning for Health Research: Still a Ways to Go.” Science Translational Medicine 13 (586): eabb1655. https://doi.org/10.1126/scitranslmed.abb1655.

Mesnard, Olivier, and Lorena A. Barba. 2017. “Reproducible and Replicable Computational Fluid Dynamics: It’s Harder Than You Think.” Computing in Science Engineering 19 (4): 44–55. https://doi.org/10.1109/MCSE.2017.3151254.

Miralles, Ignacio, Carlos Granell, Laura Díaz-Sanahuja, William Van Woensel, Juana Bretón-López, Adriana Mira, Diana Castilla, and Sven Casteleyn. 2020. “Smartphone Apps for the Treatment of Mental Disorders: Systematic Review.” JMIR Mhealth Uhealth 8 (4): e14897. https://doi.org/10.2196/14897.

Moassefi, Mana, Pouria Rouzrokh, Gian Marco Conte, Sanaz Vahdati, Tianyuan Fu, Aylin Tahmasebi, Mira Younis, et al. 2023. “Reproducibility of Deep Learning Algorithms Developed for Medical Imaging Analysis: A Systematic Review.” Journal of Digital Imaging 36 (5): 2306–12. https://doi.org/10.1007/s10278-023-00870-5.

Montgomery, Erwin B. 2024. Reproducibility in Biomedical Research: Epistemological and Statistical Problems and the Future. 2nd ed. Academic Press.

Morin, A, J Urban, and P Sliz. 2012. “A Quick Guide to Software Licensing for the Scientist-Programmer.” PLoS Computacional Biology 8 (7): e1002598. https://doi.org/10.1371/journal.pcbi.1002598.

MR, Munafò, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, Simonsohn U, Wagenmakers E-J, Ware JJ, and Ioannidis JPA. 2017. “A Quick Guide to Software Licensing for the Scientist-Programmer.” Nature Human Behaviour 1 (0021). https://doi.org/10.1038/s41562-016-0021.

National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. The National Academies Press. https://doi.org/10.17226/25303.

Niven, Daniel J., T. Jared McCormick, Sharon E. Straus, Brenda R. Hemmelgarn, Lianne Jeffs, Tavish R. M. Barnes, and Henry T. Stelfox. 2018. “Reproducibility of Clinical Research in Critical Care: A Scoping Review.” BMC Medicine 16 (1). https://doi.org/10.1186/s12916-018-1018-6.

Nosek, Brian A., and Timothy M. Errington. 2020. “What is replication?” PLOS Biology 18 (3): e3000691. https://doi.org/10.1371/journal.pbio.3000691.

Nosek, Brian A., Tom E. Hardwicke, Hannah Moshontz, Aurélien Allard, Katherine S. Corker, Anna Dreber, Fiona Fidler, et al. 2022. “Replicability, Robustness, and Reproducibility in Psychological Science.” Annual Review of Psychology 73 (1): 719–48. https://doi.org/10.1146/annurev-psych-020821-114157.

Noy, Natasha, and Carole Goble. 2023. “Are We Cobblers without Shoes?” Communications of the ACM 66 (1): 36–38. https://doi.org/10.1145/3528574.

Noy, Natasha, and Aleksandr Noy. 2019. “Let Go of Your Data.” Nature Materials 19 (1): 128–28. https://doi.org/10.1038/s41563-019-0539-5.

Nüst, Daniel, Carl Boettiger, and Ben Marwick. 2018. “How to Read a Research Compendium.” arXiv. https://doi.org/10.48550/ARXIV.1806.09525.

Nüst, Daniel, and Stephen J Eglen. 2021. “CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility.” F1000Research 10 (March): 253. https://doi.org/10.12688/f1000research.51738.1.

Nüst, Daniel, Carlos Granell, and Frank O Ostermann. 2023. “Impact of Reproducible Paper Guidelines on Computational Papers: A Longitudinal Study on the AGILE and GIScience Conference Series.” OSF. https://doi.org/10.17605/OSF.IO/XZJCH.

Nüst, Daniel, C Granell, B Hofer, M Konkol, FO Ostermann, R Sileryte, and V Cerutti. 2018. “Reproducible Research and GIScience: An Evaluation Using AGILE Conference Papers.” PeerJ 6: e5072. https://doi.org/10.7717/peerj.5072.

Nüst, Daniel, FO Ostermann, R Sileryte, B Hofer, C Granell, M Teperek, A Graser, K Broman, and K Hettne. 2018. “AGILE Reproducible Paper Guidelines.” https://doi.org/10.17605/OSF.IO/CB7Z8.

Nüst, Daniel, Frank Ostermann, Rusne Sileryte, Barbara Hofer, Carlos Granell, Marta Teperek, Anita Graser, et al. 2020. “Reproducible Publications at AGILE Conferences: Guidelines for Authors, Scientific Reviewers, and Reproducibility Reviewers.” https://doi.org/10.17605/OSF.IO/CB7Z8.

Nüst, Daniel, Vanessa Sochat, Ben Marwick, Stephen J. Eglen, Tim Head, Tony Hirst, and Benjamin D. Evans. 2020. “Ten Simple Rules for Writing Dockerfiles for Reproducible Data Science.” PLOS Computational Biology 16 (11): 1–24. https://doi.org/10.1371/journal.pcbi.1008316.

Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251): aac4716. https://doi.org/10.1126/science.aac4716.

Ostermann, FO, and C Granell. 2017. “Advancing Science with VGI: Reproducibility and Replicability of Recent Studies Using VGI.” Transactions in GIS 21 (2): 224–37. https://doi.org/10.1111/tgis.12195.

Ostermann, Frank O., Daniel Nüst, Carlos Granell, Barbara Hofer, and Markus Konkol. 2021. “Reproducible Research and GIScience: An Evaluation Using GIScience Conference Papers.” In 11th International Conference on Geographic Information Science (GIScience 2021) - Part II, edited by Krzysztof Janowicz and Judith A. Verstegen, 208:2:1–16. Leibniz International Proceedings in Informatics (LIPIcs). Dagstuhl, Germany: Schloss Dagstuhl – Leibniz-Zentrum für Informatik. https://doi.org/10.4230/LIPIcs.GIScience.2021.II.2.

Parsons, MA, RE Duerr, and MB Jones. 2019. “The History and Future of Data Citation in Practice.” Data Science Journal 18 (1). https://doi.org/10.5334/dsj-2019-052.

Peng, RD. 2011. “Reproducible Research in Computational Science.” Science 334 (6060): 1226–27. https://doi.org/10.1126/science.1213847.

Perez-Riverol, Y, L Gatto, R Wang, T Sachsenberg, J Uszkoreit, F da Veiga Leprevost, C Fufezan, et al. 2016. “Ten Simple Rules for Taking Advantage of Git and GitHub.” PLoS Computational Biology 12 (7). https://doi.org/10.1371/journal.pcbi.1004947.

Pimentel, João Felipe, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. 2019. “A Large-Scale Study about Quality and Reproducibility of Jupyter Notebooks.” In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), 507–17. https://doi.org/10.1109/MSR.2019.00077.

Pownall, Madeleine, Flávio Azevedo, Laura M. König, Hannah R. Slack, Thomas Rhys Evans, Zoe Flack, Sandra Grinschgl, et al. 2023. “Teaching Open and Reproducible Scholarship: A Critical Review of the Evidence Base for Current Pedagogical Methods and Their Outcomes.” Royal Society Open Science 10 (5). https://doi.org/10.1098/rsos.221255.

Rule, A, A Birmingham, C Zuniga, I Altintas, SC Huang, R Knight, N Moshiri, et al. 2019. “Ten Simple Rules for Reproducible Research in Jupyter Notebooks.” PLoS Computational Biology 15 (7): e1007007. https://doi.org/10.1371/journal.pcbi.1004947.

Sandve, GK, A Nekrutenko, J Taylor, and E Hovig. 2017. “Ten Simple Rules for Reproducible Computational Research.” PLoS Computacional Biology 9 (10): e1003285. https://doi.org/10.1371/journal.pcbi.1003285.

Smith, AM, DS Katz, and KE Niemeyer. 2016. “Software Citation Principles.” PeerJ Computer Science 2: e86. https://doi.org/10.7717/peerj-cs.86.

Stark, PB. 2018. “Before Reproducibility Must Come Preproducibility.” Nature 557 (7706): 613–14. https://doi.org/10.1038/d41586-018-05256-0.

Stodden, Victoria, David H Bailey, Jonathan Borwein, R. J. LeVeque, W. Rider, and W. Stein. 2013. “Setting the Default to Reproducible. Reproducibility in Computational and Experimental Mathematics.” Computational Science Research. SIAM News 46 (5): 4–6. https://stodden.net/icerm_report.pdf.

Stodden, Victoria, Jennifer Seiler, and Zhaokun Ma. 2018. “An Empirical Analysis of Journal Policy Effectiveness for Computational Reproducibility.” Proceedings of the National Academy of Sciences 115 (11): 2584–89. https://doi.org/10.1073/pnas.1708290115.

Stodden, V, and SB Miguez. 2014. “Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research.” Journal of Open Research Software 2 (1): e21. https://doi.org/10.5334/jors.ay.

Stupple, Aaron, David Singerman, and Leo Anthony Celi. 2019. “The Reproducibility Crisis in the Age of Digital Medicine.” Npj Digital Medicine 2 (1). https://doi.org/10.1038/s41746-019-0079-z.

Tenopir, C, S Allard, K Douglass, AU Aydinoglu, L Wu, E Read, M Manoff, and M Frame. 2011. “Data Sharing by Scientists: Practices and Perceptions.” PLoS ONE 6 (6): e21101. https://doi.org/10.1371/journal.pone.0021101.

Tenopir, C, ED Dalton, S Allard, M Frame, I Pjesivac, B Birch, D Pollock, and K Dorsett. 2015. “Changes in Data Sharing and Data Reuse Practices and Perceptions Among Scientists Worldwide.” PLoS ONE 10 (8): e0134826. https://doi.org/10.1371/journal.pone.0134826.

Tenopir, C, NM Rice, S Allard, L Baird, J Borycz, L Christian, B Grant, R Olendorf, and RJ Sandusky. 2020. “Data Sharing, Management, Use, and Reuse: Practices and Perceptions of Scientists Worldwide.” PLoS ONE 15 (3): e0229003. https://doi.org/10.1371/journal.pone.0229003.

The Turing Way Community. 2019. “The Turing Way: A Handbook for Reproducible Data Science.” Zenodo. https://doi.org/10.5281/zenodo.3233986.

———. 2022. “The Turing Way: A Handbook for Reproducible, Ethical and Collaborative Research.” Zenodo. https://doi.org/10.5281/zenodo.3233853.

The Turing Way Community, and Scriberia. 2024. “Illustrations from The Turing Way: Shared under CC-BY 4.0 for reuse.” Zenodo. https://doi.org/10.5281/zenodo.10556824.

Thomas Perry, Rebecca Morris, and Rosanna Lea. 2022. “A Decade of Replication Study in Education? A Mapping Review (2011–2020).” Educational Research and Evaluation 27 (1-2): 12–34. https://doi.org/10.1080/13803611.2021.2022315.

Tso, Chak Hau Michael, Michael Hollaway, Rebecca Killick, Peter Henrys, Don Monteith, John Watkins, and Gordon Blair. 2022. “The r Journal: Advancing Reproducible Research by Publishing r Markdown Notebooks as Interactive Sandboxes Using the Learnr Package.” The R Journal 14: 255–63. https://doi.org/10.32614/RJ-2022-021.

Vasilevsky, NA, J Minnier, MA Haendel, and RE Champieux. 2017. “Reproducible and Reusable Research: Are Journal Data Sharing Policies Meeting the Mark?” PeerJ 5: e3208. https://doi.org/10.7717/peerj.3208.

Vilhuber, Lars. 2020. “Reproducibility and Replicability in Economics.” Harvard Data Science Review 2 (4). https://doi.org/10.1162/99608f92.4f6b9e67.

Wilkinson, Mark D., Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, et al. 2016. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Scientific Data 3 (1). https://doi.org/10.1038/sdata.2016.18.

Wilson, C. 2021. “Impact Factor Abandoned by Dutch University in Hiring and Promotion Decisions.” Nature 595 (465). https://doi.org/10.1038/d41586-021-01759-5.

Wilson, G, J Bryan, K Cranston, J Kitzes, L Nederbragt, and TK Teal. 2017. “Good Enough Practices in Scientific Computing.” PLoS Computacional Biology 13 (6): e1005510. https://doi.org/10.1371/journal.pcbi.1005510.

Wilson, Greg. 2019. Teaching Tech Together: How to Make Lessons That Work and Build a Teaching Community Around It. Taylor & Francis. https://teachtogether.tech/.

Wood-Charlson, Elisha M., Zachary Crockett, Chris Erdmann, Adam P. Arkin, and Carly B. Robinson. 2022. “Ten Simple Rules for Getting and Giving Credit for Data.” PLOS Computational Biology 18 (9): 1–11. https://doi.org/10.1371/journal.pcbi.1010476.

Xie, Y. 2019. Bookdown: Authoring Books and Technical Documents with r Markdown. CRC Press. https://bookdown.org/yihui/bookdown/.

Xie, Y, JJ Allaire, and G Grolemund. 2018. R Markdown: The Definitive Guide. CRC Press. https://bookdown.org/yihui/rmarkdown/.

Xie, Yihui. 2015. Dynamic Documents with R and Knitr. 2nd ed. Boca Raton, Florida: Chapman; Hall/CRC. https://yihui.org/knitr/.