What to Compare a Strategy Against
“The strategy was up over the test period” is not a result. It is half of one. The missing half is the comparison — against what the same capital would have done doing something simpler, and against what a rule with no information at all would have done by luck. Until both of those exist, a backtest produces a number rather than evidence, and numbers without comparisons are how research convinces itself of things.
Educational material, not trading advice. Algorithmic crypto trading is high-risk and most retail algo traders lose money.
Two different questions
“Is this better than the obvious alternative?” and “is this better than chance?” are separate questions requiring separate machinery.
The first needs a benchmark: something a person could genuinely have done instead, held over the same period. It answers whether the complexity earns its keep.
The second needs a null model: a randomized version of your own strategy that keeps everything about it except the information content. It answers whether the rule you wrote is doing anything at all, or whether any rule with the same shape would have produced a similar figure.
Most research runs the first and skips the second, which is backwards — the null is the one that catches the failure mode you actually have.
Choosing a benchmark
The benchmark has to be something you could have held, and it has to be chosen before you look at the result. Picking it afterwards is a search, and the search selects the benchmark your strategy happens to beat.
Reasonable candidates, in rough order of how much they demand: holding the instrument the strategy trades, for the whole period. Holding an equal-weighted basket of its universe. Holding cash, which sounds trivial but is the relevant comparison for a strategy that is mostly flat. A previously validated strategy of your own — the toughest and most useful bar, because it asks whether the new idea adds anything to what you already run.
The trap in all of these is comparing at the same notional rather than at the same risk. A strategy invested all the time and one invested a third of the time are not comparable on raw return; they were not exposed to the same thing. Scale both to the same realized volatility over the test window, then compare. If the advantage disappears once exposures are matched, the “edge” was leverage — invisible in an unadjusted comparison. See position sizing and risk management basics for why exposure, not signal, dominates outcomes.
Null models: what chance looks like
A null model preserves the strategy’s mechanical shape and destroys its information. Run many of them, collect the resulting performance figures into a distribution, and see where your real result falls in it. Four constructions cover most needs:
- Random entries at matched frequency. Trade as often as the strategy did, hold for as long as it held, but choose entry times at random. This tests whether the timing rule contributed anything beyond being in the market that much.
- Shuffled signal. Keep the exact multiset of signal values the strategy generated and reassign them to random times. Same distribution of positions, no relationship to what came next.
- Sign-flipped signal. Keep the timing, randomize the direction. Useful because it separates “this rule finds moments of large movement” from “this rule finds the direction of movement” — a rule can be genuinely good at the first and useless at the second, and it will look profitable in a rising sample either way.
- Random rules from the same family. Generate rules with the same structure and randomly chosen parameters. If your tuned parameters sit unremarkably inside the distribution of arbitrary ones, the tuning found noise. This connects directly to multiple testing: why your best result is probably noise — the null distribution is, in effect, an empirical estimate of how good the best of a search should look with no edge present.
Matching the null to the strategy’s shape
Every dimension the null fails to match becomes a confound, and the confound usually favours your strategy. Match: turnover, or you are comparing a cheap thing to an expensive one; holding period, since a rule holding for weeks and one holding for minutes see entirely different amounts of drift; time in market, because in a sample that rose, anything more invested wins; gross exposure and leverage, for the same reason; the universe and the sample window, both exactly; and the cost model, applied identically — a null that trades for free against a strategy that pays is not a null, it is a handicap in your favour.
The general test: if you can name a property of your strategy that the null lacks, and that property alone could produce the difference, the comparison has isolated nothing.
Reading the comparison
Report where the real result sits in the null distribution, not merely that it exceeded the median. With enough random draws you get a percentile, and a percentile is interpretable in a way “beat the benchmark” is not.
Three habits make the reading honest. Draw enough nulls that the tail is resolved — you cannot place a result in the extreme tail of a distribution you sampled twenty times. Check the whole distribution, not just its centre: an enormous spread tells you the metric is too noisy to distinguish anything at this sample length, which is itself the finding. And adjust for your own search — if you tried many strategies, the best of them beating its null is expected, so the comparison belongs against a best-of-many null.
Look at the shape of the comparison over time, too. A strategy that beat its benchmark entirely in one short stretch and tracked it otherwise has one observation, not a track record — the same reasoning that makes a single equity curve a weak object, in one equity curve is one sample.
What comparisons cannot do
A benchmark cannot tell you the strategy will keep working, and a null cannot tell you the effect is causal. Both are tests against one sample, and both inherit its every limitation: the regimes it contains, the survivorship in the universe, the costs you assumed. A rule that clears its null on data from one regime has cleared a bar set by that regime — see regime change and why strategies decay.
What they do remove is the two most common ways of being wrong: mistaking market direction for skill, and mistaking luck for a rule. That is why the comparison is worth building before the fifth parameter, not after.
The takeaway
Every performance figure is implicitly a comparison, and if you do not state what against, you have compared it to zero — a benchmark nobody could have held and chance would beat half the time. Fix the benchmark and the null before running the test, match them to the strategy’s shape, and report the percentile rather than the verdict. A rule that survives might still fail live. A rule that does not has already failed, cheaply, which is the best outcome research can offer.