An Imperfect Guide to Imperfect Reproducibility

Gabriel Becker

Twitter: @groundwalkergmb, Github:@gmbecker


Who I Am

Gabe Becker

Twitter: @groundwalkergmb, GitHub: gmbecker

What I do

  1. Research reproducibility and related issues
  2. Contribute^ to the R language itself

^ I collaborate with, but am not a member of, the R-core development team.

Who I Was

Formerly Scientist at Genentech Research

Some content in this talk was developed (by me) while at, and is copyright Genentech, Inc. Used with permission.

What I'll Be Talking About

  • Theory - mental models
    • Reproducibility - what it does and doesn't give us
    • Generalizing Reproducibility
  • Practice - what you can do now to improve your work

I Wish Reproducibility Didn't Matter

So Many Things Would Already Be Solved

Energy (Cold Fusion)




Personalized Cancer Treatment







But We're Stuck In The Real World

And none of those claims were true^

^To the best of our current knowledge

(Computational) Reproducibility Is Not The Point

Knowledge Management




So To Recap

  • The world would be better if all results were reproducible
  • I don't really care if your results are reproducible

Seem Like A Contradiction?


What Do We Really Want?

Results Are No Use To Us If They're Not Useful


But How Does Reproducting a Result Make It Useful?


We Want

Results we can understand and feel confident using (Gavish)

  • Incorporating them into our overall understanding
  • Extending or directly utilizing them in our own work
  • Talking about them at dinner parties

1 Gavish and Donoho, A Universal Identifier for Computational Results, Procedia Computer Science 4, 2011

Trust, Verification and Guarantees

Ways We Can Trust Results

  1. Trust in Reporting - result is accurately reported
  2. Trust in Implementation - analysis code successfully implements chosen methods
  3. Statistical Trust - data and methods are (still) appropriate
  4. Scientific Trust - result convincingly supports claim(s) about underlying systems or truths

Reproducibility As A Trust Scale


Source: Gabriel Becker, copyright Genentech Inc.

What Does Strict Reproduction Prove?

It confirms^ the original analyst(s)

  1. Got the result they say they got
  2. By applying the code they provided
  3. To the input data they identified
  4. In the environment specified

And ensures you have the artifact itself

^ technically it does not confirm these in the deductive sense, rather proves them in the "beyond a reasonable doubt" sense.

A Brief Thought Experiment

Imagine an Independent, Trusted System Which

  • Accepts code and data, and environment
  • Runs the code
  • Saves and publishes results and associates them with code and env
  • Deletes the data (if necessary)

This Would Provide Guarantees

Very similar (at least) to those gained by manual, strict reproduction,

without requiring us to actually recreate the result at an arbitrary later date.

Reproducibility Isn't Everything - A Case Study

DESeq Paper


It's Reproducible!


Source: Gabriel Becker, copyright Genentech Inc.

Not The Whole Story


Source: Gabriel Becker, copyright Genentech Inc.

Really Not The Whole Story


Source: Gabriel Becker, copyright Genentech Inc.


Generalized Reproducibility

Reproducibility is great, but by itself it is neither as necessary nor as sufficient as many seem to think.

– Me (and, like, other smart people too probably)

Topics In Generalized Reproducibility

  • Comparability
  • Currency
  • Completeness
  • Provenance


Individual "Data Science" Workflow


credit: Wickham

Collaboration Is Core


Source: Gabriel Becker, copyright Genentech Inc.

Different Versions


Source: Gabriel Becker, copyright Genentech Inc.

Same Versions


Source: Gabriel Becker, copyright Genentech Inc.




Source: Gabriel Becker, copyright Genentech Inc.

Bleeding Edge Methods

  • Called this for a reason
  • Continue to be refined
  • Likely won't stay state of the art
  • May not even remain valid

No Hard-And-Fast Rule, But Generally

  • Modern versions take precedence
    • skeptical of genes in the 864 but not the 359
  • Modern methods take precedence^

^ By how much depends on why the old method fell out of favor


The Currency concern is very real in Bioinformatics, Deep Learning, and other fast moving settings. Its not really a concern when using a "classical" method (GLMs, Random Forests, etc).


Literate Statistical Practice

[The resulting document] should describe results and lessons learned … as well as a means to reproduce all steps, even those not used in a concise reconstruction, which were taken in the analysis.

– Rossini, Literate Statistical Practice (emphasis mine)





the source or origin of an object; its history and pedigree; a record of the ultimate derivation and passage of an item through its various owners

– Oxford English Dictionary

Knowing a Result's Provenance Can Help Us

  • Gain insights into the reasoning used to create it,
  • verify acceptable procedures and methods were used, and
  • know how to reproduce it.

Paraphrase of Friere et al. in Provenance for Computational Tasks: A Survey, Computing in Science & Engineering. 2008

Result Provenance Includes Input Data Provenance


Reproducibility In Practice

In theory there is no difference between theory and practice; in practice there is.

– Unattributed

A "Slightly Out-of-Order" Reproduction Process

0. No Way To Get/Recreate Code and Data?


1. Works With Modern SW Versions


2. Try To Recreate Original Environment


3. Worked In 'Original' Env, We're Good!


4. But Modern Results Disagree?


5. What Now?

We can

  1. Take the modern results and move forward using them
  2. Apply other methods to look for corroboration
  3. Throw up our hands and switch to a field that uses established classical methods

Things You Can Do Now



Scripted Analyses


Manual Steps


Which Would You Rather Be Tested On?

  1. How the chicken crossed the road, or
  2. How many licks it took

Versions and File Naming



Avoid Encoding Metadata Solely In Filenames

  • Version
    • thesis_final_revised_final_v2.Rmd
  • Species/Gene/etc
    • mydata_BRAF_mut_only.dat
  • Timestamp
    • mydata_as_of_2018_03_03.dat

Version Control


Literate Analyses

Use rmarkdown, knitr, or Sweave

This gets trust in reporting taken care of^ right away, and its super easy.

^absent actual misconduct, ie editing output files manually

Want to Use Jupytr/Rmarkdown Notebooks?

You must go watch or read Joel Grus' fantastic talk first



3 Rules If You Still Want To Use Notebooks

  1. Always run them start to finish before publication
  2. Never fully trust output in a notebook that wasn't run start to finish
  3. Always. Run. Them. Start. To. Finish. Before. Publishing.


Current State


Current State

Authors for only ~ 44% (36%, 50%) of papers in Science shared both code and data stoddenempir.png

Share Your Code!

  • If it's right no one cares how ugly it is
  • If it's wrong, readers deserve to know
    • And you do, too

Share Your Data (If Ethically Allowable)

When to share data is a tricky question.

But answers of "never" and "only once its obsolete/irrelevant" make you the villain of the piece.

Publish Science

As if you, personally, will need to understand, evaluate, use, and extend your result in 5 years, working only from published materials, having lost all personal materials.

Environment Recreation

Environment Recreation

Crucial for both multi-analyst collaboration and for strict reproduction

Remember to think about currency

Recreating/Distributing R Package Libraries

  • switchr
  • packrat
  • MRAN snapshots

Docker For Reproducibility

Its Easy



Check out

  • Makes your code runnable by others
  • Uses docker
  • Careful of changing dep version in images across time


Aspire To

  • Know where your data came from
  • Pass that information along with you results (and code)

(Even if imperfectly)

Publishing Results

What To Include

  • Code (+ detailed description of any manual steps)
  • Data (if you can)
  • Data Provenance (if you have it)
  • Version/environment info

Consider Open Access When You Can

Not a simple issue, but ask yourself how can a result be useful to people who can't even read about it?

Wrap-up and Conclusions


Think About What you Actually Want

And whether/how you can

  1. Get it
  2. Provide it to your readers

Reproducibility is NOT Valueless

But it is most usefully viewed within a larger, more nuanced context.

Share Your Code and Data (If Possible)

And don't be assholes to others who do the same. Even when you find problems in it.

Publish Science

You'd be able to trust if it came out of a lab you don't know.

Selected (and Incomplete) Further Readings


Gentleman and Temple Lang, Statistical Analyses and Reproducible Research, Bioconductor Working Papers, 2014

Marwick, Boettiger and Mullen, Packaging Data Analytical Work Reproducibly Using R (and Friends), The American Statistician, 2018

FAIR data

Wilkinson et al., The FAIR Guiding Principles for scientific data management and stewardship, Scientific Data, 2016

Dunning, de Smaele and Böhmer, Are the FAIR Data Principles fair? International Journal of Digital Curation, 2017

Case Studies and Attempts

Marwick, Computational Reproducibility in Archaeological Research: Basic Principles and a Case Study of Their Implementation, Journal of Archaeological Method and Theory, 2016

FitzJohn, Pennell, Zanne and Cornwell, Reproducible research is still a challenge, ROpenSci Blog, 2014

ROpenSci, Reproducibility In Science,

Cultural Challenges/Barriers to Reproducibilty

Basically everything Victoria Stodden has ever published.

Seriously, just go read it (at least the abstracts)

Provenance(ish) In R

Becker, Moore and Lawrence, trackr: A Framework for Enhancing Discoverability and Reproducibility of Data Visualizations and Other Artifacts in R, Journal of Computational and Graphical Statistics, 2019

Biecek and Kosiński, archivist: An R Package for Managing, Recording and Restoring Data Analysis Results, Journal of Statistical Software, 2017

Reproducible Pipelines in R

Landau, The drake R package: a pipeline toolkit for reproducibility and high-performance computing. The Journal of Open Source Software, 2018