Impact of AGILE Reproducibility Initiative

Journal of Spatial Information Science (JOSIS) · No. 32 · 2026

Improving reproducibility of GIScience publications
through novel reproducibility guidelines and revised review procedures

Carlos Granell1, Frank O. Ostermann2, Daniel Nüst3, Peter Kedron4,5, Eftychia Koukouraki6, Miguel Matey-Sanz1, Rémy Decoupes7,8, Sergio Trilles1, Anita Graser9, Tom Niers3

1Universitat Jaume I  ·  2University of Twente  ·  3TU Dresden  ·  4,5UC Santa Barbara  ·  6University of Münster  ·  7,8INRAE / Univ. Montpellier  ·  9Austrian Institute of Technology

Paper Data & Code (Github) Data & Code (Zenodo) CODECHECK Preprint Preregistration Cite
19×
higher odds of having higher rank for Data in post-intervention AGILE papers
20×
higher odds of having higher rank for Methods in post-intervention AGILE papers
55.4%
of post-intervention AGILE papers reach a High (A + O) reproducibility level for Data
0%
of GIScience papers improved significantly on Data — no formal guidelines were adopted there

TL;DR

A decade of AGILE conference papers shows that reproducibility guidelines, paired with an enforced reproducibility review, work. Comparing the AGILE conference series, which introduced author guidelines and a badge-awarding Reproducibility Committee in 2020, with the GIScience conference series, which adopted no such policy, this study finds that AGILE papers became dramatically more reproducible after the intervention (odds ratios of 12–20× across criteria, all p < 0.0001), while GIScience papers improved only modestly and showed no significant gain at all for Data. Potential reproducibility scores also track real outcomes: AGILE papers rated highly reproducible were consistently awarded actual reproduction badges. The conclusion: rather than awareness alone, clear and enforced guidance is what moves the needle.

The Problem

Reproducibility guidelines exist — but do they actually work?

Across disciplines, researchers routinely fail to reproduce published computational results, even when authors claim to share code and data. The GIScience discipline is no exception: our earlier work separately assessed the AGILE and GIScience conference series and found that reproducibility and replicability had not been a core concern in either community. That diagnosis led to a concrete intervention — the AGILE Reproducible Paper Guidelines (2019, revised 2020) — and a Reproducibility Review process to support actual reproduction of every accepted full paper and award a badge on success. The GIScience conference series adopted no equivalent policy, giving the authors an untreated comparison group.

Why this matters. If such policies are ineffective, they represent a significant waste of author and reviewer time. If they work, they offer other communities — including journals like JOSIS — a validated, transferable template for improving open and reproducible research practice.

The study is framed around three questions:

  1. RQ1 — Did AGILE’s potential reproducibility improve after the guidelines and review were introduced?
  2. RQ2 — Did GIScience’s potential reproducibility change over the same period, absent any intervention?
  3. RQ3 — Is there an observable difference between the two conference series in how reproducibility changed?

Study Design

224 papers, two conferences, one intervention

The authors assembled a nine-year, balanced corpus spanning both conference series, then had it independently double-scored against a structured rubric.

  • Corpus: 224 full papers collected for both conference series; 168 eligible after excluding non-computational papers and one transitional year (2020) per conference: 109 AGILE papers (2017–19 pre- / 2021–24 post-intervention) and 58 GIScience papers (2016 & 2018 pre- / 2021 & 2023 post-guidelines).
  • Assessors: eight recruited reviewers; every paper independently double-scored, with disagreements resolved through documented discussion.
  • Rubric (UDAO scheme): four ordinal levels — Undocumented, Documented, Available, Open (openly archived with a DOI) — applied to three criteria: Data , Methods, Results, plus a binary check for Computational Environment documentation.
  • Analysis: cumulative-link mixed-effects models comparing pre- vs. post-intervention papers, with authors treated as random effects to account for researchers publishing across both periods.

Main Findings

Guidelines plus review measurably improved AGILE papers

  • AGILE’s reproducibility jumped after the intervention. Pre-intervention, 47% of papers left their data completely Undocumented and none reached Open on any criterion. Post-intervention, over half of papers reached a High level (Available/Open) for Data, ~70% for Methods, and ~54% for Results — odds ratios of 19.3×, 20.5×, and 12.3× respectivelly, all p < 0.0001.
  • GIScience improved too, but far less — and not on data at all. Methods (OR ≈ 8.4) and Results (OR ≈ 5.3) improved significantly without any formal policy change, but the odds ratio for Data was 2.0 and not statistically significant (p = 0.068): roughly half of post-guidelines papers still left their data Undocumented.
  • AGILE’s gains are larger and broader than GIScience’s across every criterion. In percentage-point terms, the share of High papers (Available/Open) rose by +33.6 (Data), +58.9 (Methods), and +45.7 (Results) at AGILE, versus −3.9, +42.9, and +25.0 at GIScience — evidence that clear, enforced guidance is a stronger driver of change than general community awareness alone.
  • Potential reproducibility predicts real reproduction outcomes. Cross-checking rubric scores against actual AGILE Reproducibility Committee badges: papers rated “All High” earned a badge in ~51% of cases and largely succeeded, while papers rated “All Low” earned one in only approximately ~5% of the cases — the rubric is a workable proxy for real reproducibility.
  • Data is the community’s persistent weak spot. Across both conferences and both periods, Data was consistently the most contentious criterion for assessors, and the criterion least likely to achieve the Open level: technically Available datasets were often too poorly documented to be reused.

What Worked

Enforced review, not awareness alone, is what moves the needle

Both conference communities experienced the same decade of growing Open Science awareness. Only AGILE combined that awareness with two concrete mechanisms: a mandatory Data and Software Availability (DASA) section in every submission, and a CODECHECK-style reproducibility review, in which specialised “codechecker” reviewers independently attempt to execute each accepted paper’s workflow and award a timestamped, public reproducibility badge.

  • Mandatory disclosure changes author behaviour. Requiring authors to state where data, code, and methods live, or justify why they can’t, appears to close much of the “Undocumented” gap that persists at GIScience.
  • Independent execution checks catch what disclosure alone misses. The badge-vs-score comparison shows real reproduction attempts validate the rubric’s High/Low distinction, giving authors and readers a credible, verifiable signal.
  • Community spill-over is real but secondary. Some of GIScience’s modest gains may reflect authors who also publish at AGILE bringing better habits with them, but this informal diffusion is far weaker than a directly enforced policy.

Conclusion

Can a small research community shift its Open Science culture on its own?

Yes — with the right combination of policy and enforcement.

A modestly sized conference community demonstrably raised the reproducibility of its published work by pairing clear author guidelines with a lightweight, enforced reproducibility review. The underlying rubric, review process, and CODECHECK approach are largely discipline-agnostic, offering other conferences and journals a validated template to adopt. At the same time, Available is not the finish line: links and repositories decay over time, so the real target for durable reproducibility is Open and permanently archived (DOI-backed) resources.

Data Access

Everything is open

Data, code, notebooks (R/Quarto and Python/Jupyter), the Assessment Protocol, and the preprint are all openly available:

Citation

Please cite

@article{granell2026improving,
  title   = {Improving reproducibility of GIScience publications through
             novel reproducibility guidelines and revised review procedures},
  author  = {Granell, Carlos and Ostermann, Frank O. and N{\"u}st, Daniel and
             Kedron, Peter and Koukouraki, Eftychia and Matey-Sanz, Miguel and
             Decoupes, R{\'e}my and Trilles, Sergio and Graser, Anita and Niers, Tom},
  journal = {Journal of Spatial Information Science},
  number  = {32},
  pages   = {67--93},
  year    = {2026},
  doi     = {10.5311/JOSIS.2026.32.591}
}

This page was built on the Academic Project Page Template, which itself is based on the Nerfies project page. Released under a CC BY-SA 4.0 license.