A Reproducible Workflow for Strategy Research

A strategy result is only worth something if you can regenerate it, and most research setups quietly can’t. The dataset was edited in place, the parameters lived in a notebook cell that has since been overwritten, the library version has moved on, and the number of variations tried before this one is unrecorded. Reproducibility is not administrative tidiness here — it is what makes a result evidence rather than an anecdote, because the count of attempts is part of the result.

Educational material, not trading advice. Algorithmic crypto trading is high-risk and most retail algo traders lose money.

Raw data is immutable

Nothing may ever modify a file in the raw layer. Fetched data lands once, unmodified, and every transformation writes to a separate derived layer. Three tiers is enough:

data/
  raw/         # exactly what the source returned, never edited
  interim/     # validated, deduplicated, resampled
  features/    # derived inputs the strategy consumes

The rule that enforces it: derived layers are always regenerable by rerunning code against raw/, and are therefore always safe to delete. If deleting features/ would lose something, a manual edit has crept in and the pipeline is no longer reproducible.

Two additions make this much stronger. Record a content hash of every raw file in a manifest, so you can prove a result was computed against the bytes you think it was. And record the fetch parameters and timestamp alongside the data — with an API, the same request made later returns different data, which is precisely the thing a hash makes visible instead of silent.

Configuration is data, not code

Every run is defined by a config file, and the config is saved with the output. A run identified by “the notebook as it was on Tuesday” is not identified at all. A config that names the universe, date range, bar interval, feature parameters, cost assumptions, and validation scheme can be diffed, versioned, and re-executed.

runs/
  2026-07-24T0913-ab3c1f/
    config.yaml        # everything needed to reproduce
    metrics.json       # what came out
    trades.parquet     # every fill
    env.txt            # pinned dependency versions
    git.txt            # commit hash + dirty flag

The directory-per-run pattern costs nothing and answers the question that always comes up months later: what exactly produced this number? Recording the git commit and whether the tree was dirty matters more than it sounds — a result generated from uncommitted changes is not reproducible, and knowing that is better than assuming otherwise.

Determinism

Seed everything, and be aware of what a seed doesn’t cover. Any random split, bootstrap, or stochastic optimizer takes a seed, and the seed belongs in the config rather than in a global call somewhere in a notebook.

Sources of nondeterminism that survive seeding: dictionary or set iteration order feeding into a numerical result, parallel reductions whose floating-point summation order varies, and multi-threaded library backends. Perfect bit-level determinism is often not worth chasing; what matters is that reruns agree to a tolerance you have actually checked, so that a difference between two runs means a change and not noise.

The related discipline is to make the pipeline runnable end to end from a single command. If reproducing a result requires executing seven notebook cells in the right order, it will not be reproduced.

Notebooks at the edges only

Notebooks are excellent for looking at things and terrible as the place logic lives. Hidden execution order, stale state from deleted cells, and no meaningful diff make a notebook the least reproducible artefact in the stack.

The workable division: functions and pipelines live in importable modules under version control and are called from notebooks; notebooks contain plots, tables, and commentary. When a notebook cell becomes something you rely on, promote it to a module and import it back. This also makes the code testable, and a few unit tests on feature computation catch a surprising share of the leaks described in auditing your code for lookahead bias — an assertion that a feature at time t is unchanged when future rows are altered is a cheap, decisive test.

Pin the environment

A silent library upgrade changes results. A default argument changes, an aggregation’s tie-breaking changes, a numerical routine’s precision changes — none of it announced, all of it capable of moving a metric.

Use a virtual environment per project, pin exact versions in a lockfile, and store the resolved versions with each run. A container earns its keep for research revisited across years, because a lockfile only pins what the package manager can still find.

The research log

The most valuable artefact in the whole setup is a plain record of every configuration tried, written before the outcome is known. One row per attempt: date, hypothesis in one sentence, config path, result, and a verdict.

Its purpose is not tidiness. It is that the number of attempts determines how impressed you should be by the best of them. A strategy that is the best of three tested ideas and one that is the best of two hundred deserve entirely different levels of confidence, and without a log the count is unknowable — the failures are exactly what memory discards. This is the practical countermeasure to the problem set out in multiple testing and strategy selection.

Writing the hypothesis first has a second benefit: it makes post-hoc rationalization visible. If the logged reason for the test doesn’t match the story you now tell about why it works, the story was invented after the result, which is the signature of the trap in how overfitting hides in a trading strategy.

Validation belongs in the config

How a result was validated should be a recorded parameter, not a habit. Whether the run used a single holdout or walk-forward, the window lengths, the embargo, the cost assumptions — all of it in the config, so a reader can see the validation scheme without reading the code. See how walk-forward validation works.

The same applies to the data-quality report: row counts, gap statistics, synthetic-row fraction, and flagged outliers stored with the run, per data quality checks for crypto price history. A metric without its data report is uninterpretable, because you cannot tell whether it describes the market or the loader.

Why the effort pays

The immediate return is smaller than the eventual one, which is why this is usually skipped. The eventual returns are concrete: you can re-run a six-month-old result to check whether a bug you just found affected it; you can tell whether a change improved things or just moved noise; you can extend a study rather than rebuilding it; and — the big one — you can honestly count your attempts, which is the only defence against the failure mode that ruins most strategy research.

For where these practices sit in the wider toolkit, see the Python tooling stack for crypto quant research. The libraries are interchangeable. The discipline is what makes their output mean anything.