Testing a Backtester With Data You Made Up

You will test a strategy dozens of times before you trust it. The engine that produced those tests usually gets tested never — and it sits between every hypothesis you have and every number you read. A bug in the strategy costs one wrong idea. A bug in the engine costs every conclusion you have ever drawn with it, including the ones you rejected.

Educational material, not trading advice. Algorithmic crypto trading is high-risk and most retail algo traders lose money.

Real data cannot test an engine

With real prices you do not know the correct answer, so a bug in the engine is indistinguishable from a finding. If your accounting double-counts a fee, the strategy looks slightly worse and you shrug. If your bar alignment is off by one in the flattering direction, it looks slightly better and you get excited. Neither shows up as an error, a warning, or a crash. It shows up as a result.

Synthetic data breaks the symmetry. Construct a price series simple enough that you can compute the correct profit and loss on paper, and any discrepancy is unambiguously a defect in your code — no interpretation required.

Series with answers you can compute by hand

Build a handful of degenerate series, each of which pins down one behaviour.

A flat line. A constant price, traded by any strategy at all. Gross profit and loss must be exactly zero, and net must be exactly the cost model’s output for the turnover generated. This catches a startling proportion of accounting bugs, because it removes market movement and leaves only your bookkeeping.
A single step. Constant, then one jump, then constant. A strategy long across the step captures it exactly; a flat one captures nothing. This isolates when your engine believes a position exists, which is where off-by-one alignment lives.
A straight ramp. A price rising by a fixed increment per bar. A rule long throughout must capture the total rise minus costs, to the last decimal. If it captures slightly less, you are probably paying for the position rather than for the change in it.
A sawtooth. A regular oscillation with known amplitude and period. A mean-reversion rule tuned to that period should capture close to every leg. Anything less is a signal-timing or fill-timing bug, and the two are distinguishable by whether the shortfall repeats on every leg.
A pure random walk with no drift. Expected gross performance is zero for any rule using only the past. Any strategy that reliably makes money here is looking at the future.

That last one deserves emphasis: it is the cheapest lookahead detector in existence, it takes four lines, and it finds the leak without you having to suspect it first. The audit in auditing your code for lookahead bias is where you go next.

price = pd.Series(100.0 + np.arange(n) * step)     # ramp
res   = engine.run(strategy=always_long, price=price, fee=0.0)
assert abs(res.pnl - (price.iloc[-1] - price.iloc[0])) < 1e-9

The numbers in that sketch are arbitrary inputs chosen to make the arithmetic obvious, not values with any meaning.

The bugs this actually catches

In practice these tests fail for a small and repeating set of reasons.

Off-by-one bar alignment — the position is credited from the bar the signal was computed on rather than the next one, so the engine captures a move it decided on after seeing. Cost charged on position instead of on the change in position, which makes every holding period look worse than it is and pushes a parameter search toward higher turnover to escape a phantom penalty. Sign errors on short exposure, usually in only one of the several places shorts are handled. Summing returns where you should compound them (or the reverse), invisible over short samples and growing with test length. State leaking between runs, so the second backtest in a sweep starts from the first one’s balance — which makes an entire grid subtly wrong in a way no individual result reveals.

Each item on that list corresponds to one of the degenerate series above, which is what makes it a usable checklist.

Differential testing against a dumb engine

Write a second engine that is too simple to be wrong, and require the two to agree. A plain loop over bars — one position variable, one cash variable, no vectorization, no cleverness, no performance concerns — is perhaps thirty lines and can be read in full by a human. It will be far too slow for research; its job is to be obviously correct on small inputs.

Then assert that the fast engine and the slow one produce the same equity curve, to floating-point tolerance, on random inputs. Disagreement means one is wrong, and because the slow one is auditable you can find out which. This is the highest-value test in the set, and the practical answer to the auditability question in how to evaluate a backtesting library: if you cannot reimplement an engine’s accounting simply enough to check it, you cannot verify its output at all.

It is also how to use both execution styles together rather than choosing between them, as in event-driven vs. vectorized backtests — the event-driven implementation becomes the reference the vectorized one is checked against.

Properties that hold for every input

Some invariants need no specific series, which makes them cheap to test against random data.

A strategy that never trades produces a constant equity curve, whatever prices do.
Multiplying every price by a constant leaves percentage returns unchanged. If it doesn’t, something is working in absolute units where it should be relative.
Reversing the sign of every position reverses gross profit and loss exactly. Costs, being unsigned, should be identical.
Total cost is monotone in turnover: more trading never costs less.
Adding a bar to the end of the data cannot change any earlier value of the equity curve. A violation here is a lookahead bug with no hiding place.

Generating random price paths and asserting these properties across many of them finds edge cases you would not think to construct — empty positions, immediate reversals, zero-volume bars.

Where the tests live

In the test suite, running before any research run, not in a notebook you executed once in March. Engine tests are ordinary software tests with an ordinary home: fast, deterministic, seeded, automatic. Determinism matters more here than elsewhere, because a flaky engine test is worse than no test — it trains you to ignore failures. This is the same discipline as the rest of the setup in a reproducible workflow for strategy research: if validation is part of the configuration rather than something you remember to do, it happens.

The takeaway

Strategy research is unusual in that the measuring instrument is code you wrote yourself, changed last week, and never calibrated. Synthetic data is how you calibrate it: construct the cases where you already know the answer, and let the engine prove it reproduces them. The effort is repaid the first time a promising result turns out to be a bug — because the alternative is finding out from your fills instead.