Recommendations and practices for open and reproducible research
As a reviewer or researcher interested in the work of others, if you were to reproduce a study, you would
read the published manuscript (Keshav 2007) under the lens of the assessment criteria for reproducibility (Nüst, Boettiger, and Marwick 2018)
look for published data and code
recreate the analysis in your execution environment
compare the results you obtain to the published results
if they are the same, the study is reproducible; otherwise, it is not
Very cool; But If you had to write a reproducible paper… How do you start?
We have heard many excuses to avoid taking reproducible practices seriously in one’s research, such as “I have never heard of reproducibility before”, “I’m not good at using tools; Did you say github and docker?”, “I have never been trained”, “I don’t know where to find tutorials, guidelines…”, “I’m superbusy! I do not have enough time”, and “I do theoretical stuff. Reproduction is not for me”
No matter your excuse; If you do not want to fool yourself, reproducible research practices are essential to enable not only reproducible but transparent, open and honest science. Indeed, conducting reproducible research is neither overly difficult nor does it require encyclopedic knowledge of multiple research tools, standards, and protocols. The good news is that most researchers already do much of the work necessary to make research reproducible. The point here is then to identify those aspects that could improve the level of reproducibility of your research and adapt them to your workflow and research habits.
You should ask yourself:
What habits should I adopt to change my daily routine?
What aspects should I improve the most?
What guidelines or recommendations can I follow?
Key guidelines
There are excellent practical guides with general recommendations for promoting reproducibility, research data management and open science. As an author, who wants to integrate reproducibility research practices into her next research paper, the following resources are an excellent starting point:
The British Ecological Society publishes brief guidelines for conducting open science on ecology but applicable to any discipline. Among them, the guide Reproducible code (Cooper and Hsing 2019) explains organizational and managerial aspects for making software code more reproducible.
Passport for Open Science: A Practical Guide for PhD Students (Berti et al. 2022) explains how the principles of open science can be applied to doctoral research. The proposed practices are applicable to any discipline and, although focused on doctoral students, they are aimed at any researcher regardless of their previous experience.
A Beginner’s Guide to Conducting Reproducible Research (Alston and Rick 2021) is a very concise, practical guide to apply reproducible practices to a research project.
The Turing Way is indeed a compendium of guides on open science, reproducibility, and research ethics, among other interesting topics. Have a look at the Guide for Reproducible Research (The Turing Way Community 2022).
Reproducible Publications at AGILE Conferences: Guidelines for Authors, Scientific Reviewers, and Reproducibility Reviewers (Nüst et al. 2020) describes specific aspects to write conference papers in the field of GIScience.
Before, during and after
flowchart TB A[Before data analysis] --> B[During data analysis] B --> C[After data analysis]
(Alston and Rick 2021) outlined basic steps toward making research more reproducible base on three stages of a research project: (1) before data analysis, (2) during analysis, and (3) after analysis.
We borrow this simple outline to better categorise and explain the set of practical tips and recommendations that follow. Note that most of the following recommendations are already widely accepted best practices for scientific research and that striving for a reasonable level of reproducibility is more achievable than you may expect. Essentially, researchers must think carefully before, during, and after analysis to ensure the reproducibility, openness and transparency of their work.