flowchart LR A[BEFORE ANALYSIS] --> B[DURING ANALYSIS] B --> C[AFTER ANALYSIS]
9 Recommendations and practices: before, during and after
Researchers and scientists in academic institutions are expected to publish at a regular rate. The more, the better. It may not be demanded, but scientific publications certainly give a boost to a researcher’s carreer (Szabo 2025). Incentives do not necessarily align with the goals of reproducibility, openness, and transparency. For example, the “publish or perish” culture encourages researchers to focus on quantity over quality, leading to rushed publications that may lack thorough documentation and sharing of data and code (Figure 9.1).
It is crucial to adopt practices that promote reproducibility at every stage of the research process, from planning to execution to dissemination. In this regard, (Alston and Rick 2021) outlined a series of basic actions and recomentdatiosn toward making research more reproducible based on three stages of a research project: (1) before, (2) during, and (3) after analysis.
We borrow this simple outline to better categorise and explain the set of practical recommendations and practices that follow. Note that most of the following recommendations are already widely accepted best practices for scientific research and that striving for a reasonable level of reproducibility is more achievable than you may expect. Essentially, researchers must think carefully before, during, and after analysis to ensure the reproducibility, openness and transparency of their work.
9.1 Before analysis: planing for reproducibility
“Science is a way of trying not to fool yourself. The principle is that you must not fool yourself, and you are the easiest person to fool.” (Quotes)
Nobel laureate and physics professor Richard Feynman
This slide deck, titled “Before analysis: planning for reproducibility” focuses on the organisational steps a researcher must take before a single line of code is written or data is collected. The central thesis is that reproducibility is not something you “fix” at the end of a research project; it is a mindset that must be integrated into the planning phase to avoid the “reproducibility debt” that accumulates through disorganised workflows.
The presentation emphasises that the “research journey” is often messy and non-linear. To counter this natural entropy, researchers should adopt a “project-centric” approach. This involves treating every research project as a self-contained unit where data, code, and documentation are inextricably linked. It highlights that the primary beneficiary of these practices is often your “future self”, who will inevitably forget the logic behind current decisions. It advocates for moving away from “hand-crafted” manual processes toward automated, scripted workflows that ensure the path from raw data to final results is transparent and repeatable.
List of recommendations/practices:
[#1] Adopt a project-oriented workflow: Use dedicated folders to ensure that all paths are relative to the project root, making the project portable across different computers. In short, one project lives in one folder. The following question is to define what a project is. This usually depends on one’s type of work and circumstances, but some examples of a project are: an experiment in a PhD project, a master’s thesis project, ideas for future research, regular meeting notes/minutes, teaching materials, a review paper, a conference presentation, workshop or seminar materials, a book, or a PhD thesis manuscript. This point is to keep all of the related resources to a given project in its dedicated folder.
[#2] Organise files logically: Choose a folder structure that is consistent, informative, and works for you — and then stick to it. A typical layout separates documentation files (
README.md,LICENSE,CODE_OF_CONDUCT,CONTRIBUTING) from data (data,data-raw), code (scripts,analysis), results (reports,figs), and other documentation (notes,docs). There is no single “correct” structure, but established templates such as The Turing Way’s project template offer a solid, well-documented starting point.[#3] Use consistent, machine-readable file names: Do not rely on filenames like
final_v2_REALLY_FINAL.docx. File names should be machine-readable, human-readable, and play well with default alphabetical ordering. Prefix script names with a number or letter that reflects the analysis sequence (e.g.,01_download_data.R), and prefix data file names with an ISO 8601 date (YYYYMMDD_survey.csv) so files sort naturally by time and their processing order is immediately visible.[#4] Separate raw data from processed data: Never overwrite or otherwise touch your raw data. Store it permanently, read-only, in a dedicated subfolder (e.g.,
data-raw), echoing the lesson of Newton’s letter to Flamsteed (Noy and Noy 2019). Use scripts, not manual edits, to clean and process it, saving the results in a separate subfolder (e.g.,dataordata-clean). Document the cleaning process — main steps, diagrams, and the content, structure, and provenance of each dataset — in a plain-textREADMEfile.[#5] Leverage open data formats: Favour open, text-based formats whenever possible, since they are independent of any specific software vendor and remain accessible over the long term. When a proprietary format is unavoidable, provide an open equivalent alongside it; for instance, CSV next to an Excel file, or GeoPackage next to an ESRI Shapefile. Repositories such as DANS (the Dutch national centre of expertise and repository for research data) publish guidance distinguishing preferred from non-preferred formats to help you choose wisely.
[#6] Document and write “README” files early: Create a
README.mdfile at the start of the project, in the root folder, to describe the project’s purpose, structure, and requirements. This serves as a guide for both current and future collaborators, including your future self. On platforms like GitHub, this file renders automatically when written in Markdown. Add furtherREADMEfiles in subfolders as needed to document metadata or complex content, and keep anotesfolder to track ideas, discussions, and decisions as the project evolves.[#7] Plan for data sharing: A license is a contract between authors and users (Jolly et al. 2012); without one, copyright is automatically and fully retained, which prevents others from legally reusing your data. If you intend to make your data public, always attach a license file (
LICENSE.mdorLICENSE.txt), choosing among options such as Creative Commons (BY, SA, NC, ND combinations), CC0 for public-domain dedication, or Open Data Commons licenses. Institutions often provide their own recommendations — UJI, for example, suggests CC BY-SA for theses and CC BY-NC-SA for teaching materials.[#8] Plan for code sharing: As with data, any software you plan to release publicly needs an explicit license, since unlicensed code defaults to full copyright and cannot legally be reused by others. Broadly, code licenses fall into two families (Morin et al. 2012): permissive licenses, which only require attribution and are generally recommended for academic work, and copyleft licenses, which require derivative works to carry the same license as the original. Choosing the right one early avoids ambiguity for anyone who later wants to build on your code.
[#9] Use version control: Turn your local project folder into a version-controlled repository to track how your work changes over time and to roll back to earlier versions when needed. Version control systems such as Git are especially suited to plain-text formats (code, Markdown, and other text documents) rather than rich or binary formats like Word files or images. Think of it as a “time machine” for your code (and other text files, data files, etc.), one that also documents the reasoning behind each change through commit messages.
[#10] Use online remote repositories: Pairing your local Git repository with an online hosting service such as GitHub, GitLab, or Bitbucket adds collaborative features and development support on top of version control itself, making it far easier for individuals and teams to share and jointly develop code. Beyond hosting, these platforms integrate with issue tracking, pull requests, and continuous integration, and templates like Cookiecutter Data Science offer ready-made repository structures for reproducible projects.
NoteSlide Deck
9.2 During analysis: reproducible workflows
“The ideas we can most trust are those that have been the most tried and tested.” (Quotes)
Philosopher of science Karl Popper
This slide deck, titled “During Research: reproducible workflows”, focuses on the execution phase of a project. While the previous stage (“Before”) dealt with setup and preperatory actions, this one provides the practical habits required to maintain a reproducible record as the research actually happens.
The core idea is that reproducibility is a continuous process, not a final step. Here, the concept of the “computational environment” is central, sice your results are a product of not just your data, but the specific versions of software, libraries, and operating systems you use in a research project.
The deck also warns against “manual interventions” (like fixing a typo in a CSV file by hand), as these create “dark matter” in the research process, that is, steps or actions that happened but left no record. Instead, the following set of recommendations advocate for a script-based workflow where every transformation of data (and optherdigitla resoruces) is recorded in code. Consequently, a significant part of the deck is dedicated to the concept of literate programming (using tools like Quarto), which allows researchers to weave together their narrative, their code, and their results into a single, dynamic document that automatically updates when the data changes.
List of recommendations/practices:
[#11] Open data does not guarantee reproducible data: Openness — sharing data, code, or publishing open access — is an enabler of reproducibility, but it does not by itself guarantee it (Chiarelli et al. 2021). Open data can still be poorly documented or impossible to recreate results from, and conversely, “good” work (of high academic quality) is not automatically either open or reproducible. Treat openness and reproducibility as related but distinct goals, each requiring its own deliberate effort.
[#12] Pay attention to data that is relevant for reproducibility: Reproducibility requires access to the datasets needed to recreate reported results. But access does not necessarily mean the data must be openly published, independently citable, or discoverable on its own. When a dataset is meaningless outside its accompanying article, the recommended approach is to deposit a full reproducibility package (data, code, and documentation) in a repository such as Zenodo, rather than relying on supplementary material tied only to the paper.
[#13] Adopt FAIR principles for data management with caution: FAIR data is not necessarily reproducible data. FAIR principles focus on making data Findable, Accessible, Interoperable, and Reusable, but it does not guarantee that the data is well-documented or that the analysis can be reproduced. To ensure reproducibility, you need to go beyond FAIR and also focus on the quality of documentation, version control, and providing clear instructions for how to use the data. As a rule of thumb, if a dataset is a valuable (Noy and Goble 2023), i.e. reusable contribution in its own right, invest the extra time to make it fully FAIR; if data only supports one specific paper, a
READMEfile, a license, and a Zenodo deposit are enough.[#14] Use open source tools whenever possible: Prefer open source alternatives to proprietary software wherever feasible — QGIS instead of ArcGIS, OpenStreetMap instead of Google Maps/Places, and Python or R instead of Stata, SPSS, or Excel. Open tools are free from vendor lock-in, and more accessible to collaborators and readers who may not hold a commercial license. Where a fully open workflow isn’t practical, a mixed approach — such as ArcGIS’s Python notebooks — can still turn implicit, click-based analysis into explicit, code-based analysis.
[#15] Learn/use scripting languages: Avoid manual “point-and-click” operations in software like Excel. If a change is made to the data, it should be documented in a script. Scripting languages such as R and Python let you play with data, measure, and explore it while remaining appropriately distrustful of your own intuition, since every step of the analysis is described explicitly rather than hidden behind clicks. This lets others understand not just what was done but exactly how — embodying the principle “show me, not trust me” (Stark 2018). Today, AI assistants can be used to generate code snippets, but the researcher should understand and review the code to ensure it is correct and reproducible.
[#16] Embrace notebooks: Use Quarto, marimo or similar tools to keep your text and code (scripts) in the same file. This prevents, for example, “copy-paste” errors between your statistics software and your word processor. A computational notebook is an editor made up of cells that can each hold narrative text, executable code, or the resulting output (charts, tables, plots, maps), enabling literate programming. Notebooks are increasingly treated as first-class scientific outputs — from peer-reviewed notebook submissions at EarthCube and AGU’s Notebooks Now! to theses written as notebooks — though studies show many published notebooks fail to run or reproduce their original results, underscoring the need for good notebook practices (Pimentel et al. 2019).
[#17] Manage software dependencies: Save information about your software versions and package dependencies. This allows others to recreate the exact technical conditions under which your results were generated. Tools like
renvfor R orvenvfor Python let you snapshot and later restore the precise package versions a project depends on, so a collaborator (or your future self) can reconstruct the same environment rather than fight version mismatches. Pair this with other good coding practices, such as setting a random seed, to further stabilise your results.[#18] Record your computational environment: Beyond managing individual package versions, containerisation tools capture an entire computational environment — operating system, libraries, and configuration — as a reusable snapshot (Nüst, Sochat, et al. 2020). Docker and related tools are the most widely used option, while services like Binder and
repo2dockercan automatically build and deploy a cloud-based container directly from a Git repository’s configuration files. This guarantees that an analysis runs identically regardless of who executes it or when.[#19] Adopt FAIR principles for software/code management: Just as data can be made FAIR, source code should be documented (Barker et al. 2022) so that it is Findable (deposited in a repository providing DOIs and metadata), Accessible (downloadable), Interoperable (built with open source languages and software), and Reusable (released under an permissive/open license). Researchers should also cite software properly, treating it as a scholarly output in its own right rather than an invisible tool.
[#20] Automate workflows: Use tools like the venerable GNU Make or modern alternatives (e.g., python-based
Snakemake) to automate the execution of your analysis pipeline. This ensures that all steps are executed in the correct order and that any changes to the data or code will trigger the necessary updates to the results. GNU Make, despite being over 40 years old, remains a relevant way to coordinate a series of otherwise independent scripts into a single “one-entry-point” pipeline.
NoteSlide Deck
9.3 After analysis: writing and sharing reproducible resources
“I do not mind if you think slowly, but I do object when you publish more quickly than you think.” (Quotes)
Nobel laureate and physics professor Wolfgang Pauli
The third and final presentation, titled “After analysis: writing and sharing reproducible resources,” focuses on the final stages of the research lifecycle: how to document, cite, and share materials so that others (and your future self) can verify and build upon your work.
The core message is a shift from “Trust Me” (results published without access to data/code) to “Show Me” (pre-publication transparency) (Stark 2018). It argues that the common phrase “data and code available upon request” is insufficient and often leads to a dead end. Instead, reproducibility requires proactive sharing of all computational artifacts, including software versions, specific libraries, and raw data, at the time of submission.
The deck emphasises that software is a primary research object, not just a tool. This means researchers must explicitly cite the software and packages they use, just as they would cite a journal article. The presentation advocates for the use of Research Compendia and modern scientific and technical publishing tools like Quarto to create “dynamic documents” where analytical code and narrative are linked. The ultimate goal is to move towards “reproducible manuscripts” that allow readers to interact with the data and code directly within the published article, fostering a culture of openness and verification in scientific research/publishing.
List of recommendations/practices:
[#21] Move beyond “Available Upon Request”: Never rely on post-publication requests. Data and code must be available at the time of pre-publication or submission; again, a “show me” stance rather than a “trust me” one (Stark 2018). Consider (Stodden et al. 2018)’s study, where the authors analysed 204 computational articles in Science published under an “available upon request” policy. Upon request, they received data and/or code from authors only for 44%, and were able to reproduce the reported findings in just 26% of cases. It is improvement over having no policy at all, but still far short of what genuine reproducibility requires.
[#22] Report software versions: Always specify the exact versions of packages, libraries, and frameworks used. A change in version can lead to different results becuase differences in software versions, external libraries, or compilers can be enough to break reproducibility in some disciplines (Mesnard and Barba 2017), so recording precise version numbers for every tool used in a paper is essential context for anyone trying to recreate your results.
[#23] Cite the software you use: Treat software as a first-class citizen (Smith et al. 2016). Whether to cite every piece of software used depends on how central it was to the results, but anything key to the research findings should be cited properly, not just linked as a URL. For exampel, use
citation()in R to get proper references for the R engine and specific packages, and prefer citing a tool’s dedicated paper (e.g., SciPy) over a footnote, so that the authors of research software receive the same scholarly credit as authors of any other publication.[#24] Include a DASA Section: Add a “Data and Software Availability” (DASA) section to your paper (Nüst, Ostermann, et al. 2020). This section should provide persistent links (e.g., DOIs) to repositories and describe the conditions for access, such as copyright, licenses, or procedures for protected data. A good DASA section is concise but complete: it typically names the exact software versions used (e.g., R v4.3.0), points to the specific package versions and data sources, and links to the archived code and computing environment, so a reader can reconstruct the full technical context behind the results.
[#25] Share preprints: Deposit versions of your paper in repositories (like arXiv, BioRxiv, or SocArXiv) before formal journal review to increase visibility and access. No extra effort, it’s the same file. But the benefits are immediate because your work becomes citable before the 6–18 months of peer review. Empirically, papers with preprints receive a significant citation advantage as shown by one large 2024 study of 122,000 PLOS publications that found a ~20% citation advantage for preprint-deposited articles (Colavizza et al. 2024). Preprints are typically discipline-specific: arXiv for physics, mathematics and computer science, BioRxiv for biological sciences, EarthArXiv for Earth Science and so on. Before depositing, choose an appropriate license, since the preprint itself becomes a citable, reusable output in its own right.
[#26] Formally share data and code: Document and deposit your resources in permanent repositories (like Zenodo or Open Science Framework) and link them to your GitHub account for versioned archiving. Beyond simply depositing the files, include them as proper references in your paper and cite them, most naturally within the DASA section, so that the data and code become traceable, citable parts of the scholarly record.
[#27] Create research compendia: Organise all digital materials (data, code, reports, diagrams, etc.) in a standardised, recognisable collection to make the entire project easy to navigate. A research compendium brings together everything produced during a project — datasets, code, and other outputs such as reports or questionnaires — into one archived, well-structured package, following established principles for how such compendia should be organised. Platforms like the Open Science Framework, Octopus, and ResearchEquals are designed specifically to support sharing every step of the scientific process this way.
[#28] Use literate programming tools: Adopt tools like Quarto (Allaire 2023) or RMarkdown (Xie et al. 2018) to create dynamic documents where code and text are woven together, ensuring figures and tables update automatically. This lineage runs from
knitr(2011) through RMarkdown (2014) to Quarto (2022), each iteration expanding what a single document can produce — papers, books, websites, blog posts — while keeping the underlying code, and therefore the reproducibility of every figure and table, tightly bound to the narrative around it, whether written in R or Python.[#29] Aim for reproducible manuscripts: Look toward the future of publishing — interactive papers that allow readers to engage with the data and code directly within the article. Two threads are converging here: interactive papers, which let readers manipulate figures and explore results directly in the text, and reproducible papers, which guarantee that the underlying code and data genuinely regenerate those results. The desirable vision is publications that are both interactive and reproducible at once, rather than treating these as separate, optional add-ons to a traditional PDF.
[#30] Spread the word: Reproducibility is a community effort and a cultural change. Encourage you colleagues to adopt these practices. No single set of guidelines or tools can transform research culture because lasting change depends on researchers embrancing these practices and sharing them with colleagues, students, and collaborators, so that openness and reproducibility become the norm, rather than an exceptional effort.
NoteSlide Deck
new, original, and innovative scholarship that broadens and expands our current understanding of [put your area of knowledge here] is needed and should continue to be encouraged. Nevertheless, the emphasis on innovation over systematic replication has a cost: the inability to build cumulative knowledge and to be able to state with confidence what is, and is not, known. (Monk and Koziarski 2023), (Lowenkamp et al. 2003)
Would you reserve 20% of your weekly research time for reproducibility/replicability research activities?. I propose 20% as a minimum, but any higher percentage is valid, of course.