Multiple Testing: Why Your Best Result Is Probably Noise

If you evaluate one strategy and it looks good, that is evidence. If you evaluate two hundred and report the best one, the best one looks good largely because you evaluated two hundred. The maximum of many noisy estimates is biased upward, and the bias grows with the number of trials — which means the count of things you tried is part of your result, and omitting it makes the result uninterpretable rather than merely incomplete.

Educational material, not trading advice. Algorithmic crypto trading is high-risk and most retail algo traders lose money.

The mechanism

Every performance estimate is a true value plus noise, and searching over configurations selects for favourable noise. With one test, your estimate is unbiased: it’s as likely to understate as to overstate. Take the maximum over many tests and you have selected the estimate with the largest positive error, so the winner’s measured performance systematically exceeds its true performance.

The classic illustration: give a thousand people ten coin flips each and someone will flip nine heads. That person has no skill, and their record is not evidence of any. Report only them and you have manufactured a champion out of nothing.

The strength of the effect depends on two things: how many independent trials you ran, and how noisy each estimate is. Crypto strategy research is unusually exposed on both counts. Parameter sweeps make thousands of trials trivial, and short histories of an extremely noisy asset class make each individual estimate wide. Those two facts multiply.

Counting trials honestly

The number that matters is not the number of backtests you ran today; it’s the total number of distinct configurations you have ever evaluated on this data. That includes:

Every cell of every parameter sweep. A grid of ten lookbacks by ten thresholds is a hundred trials, not one.
Every bar interval you tried and abandoned.
Every universe variation — asset sets, liquidity filters, exclusions.
Every metric you looked at before settling on one to report.
Every earlier version of the idea, including the ones you modified after seeing them fail. Retrying a rejected hypothesis with a tweak is a parameter sweep conducted across weeks, with your memory as the search algorithm.
Everything anyone else has published about the same data. The literature on well-known effects is itself the product of a vast unrecorded search, which is one reason widely documented anomalies so often fail to replicate out of sample.

The last two are why an honest count is essentially impossible to obtain after the fact, and why the only workable answer is to keep a log written before outcomes are known — see a reproducible workflow for strategy research.

Note also that trials are rarely independent: neighbouring parameter values produce correlated results, so a hundred-cell grid is worth substantially fewer than a hundred independent trials. That cuts the correction down, but not to nothing, and the effective number cannot be computed precisely — treat it as an order-of-magnitude adjustment, not a formula.

Formal corrections and their limits

Statistics offers ways to adjust for multiplicity, and all of them require knowing the trial count.

The simplest is a Bonferroni-style adjustment: divide your significance threshold by the number of tests, or equivalently demand a proportionally more extreme result. It is crude and conservative, and it assumes independence, which correlated parameter grids violate — but as a sanity check it is better than nothing.

False discovery rate procedures are less conservative and control the expected proportion of reported findings that are false rather than the chance of any false finding. This is often a better match for research where you expect some real effects among many candidates.

In the quantitative finance literature there are also purpose-built adjustments — a deflated Sharpe ratio that discounts an observed Sharpe for the number of trials, the sample length, and the non-normality of the returns, alongside estimates of the minimum backtest length needed before a given ratio means anything. The specific formulas matter less than the conclusion they all reach: the threshold for “impressive” rises sharply with the number of configurations searched, and with short samples it rises to levels most research never reaches.

None of these fix the deeper problem, which is that the true trial count includes decisions you made before you started counting: the choice of asset class, of data source, of the general idea. No correction can adjust for a search you didn’t record.

Practical countermeasures

Since the correction can’t be computed exactly, the answer is to reduce the search and to raise the bar.

Form the hypothesis first. A test with a stated reason before it is run is worth far more than the same test found by scanning. If you cannot say why the effect should exist without referencing your backtest, it was found by search.
Prefer few parameters. Fewer axes means fewer trials means less selection bias. A two-parameter rule with a rationale beats a nine-parameter one that happens to align — see how overfitting hides in a trading strategy.
Judge the whole surface, not the peak. Report the median result across a parameter grid, not the maximum. If the median is unattractive, the peak was noise. A broad plateau is believable; a single spike is not.
Reserve a final holdout you look at once. Once. Looking twice makes it part of the search, and there is no way to un-look.
Demand more from a heavily searched result. If the best of a hundred configurations is only slightly better than the median, treat it as noise regardless of its absolute value.
Track out-of-sample results as their own sample. Every strategy you deploy is a forward test. Recording all of them — including the ones you abandoned — is the only genuinely unbiased performance record you will ever have.

Why this is the hardest bias to fix

Lookahead bias is a bug you can find by reading code. Survivorship bias is a data problem you can fix with a better universe. Multiple testing is neither — it is a property of the process, and it leaves no trace in any artefact you can inspect. A walk-forward-validated strategy, correctly implemented on clean point-in-time data with honest costs, can still be pure noise if it was the best of a thousand attempts, and nothing in the output will indicate that. See how walk-forward validation works for what that validation does and does not cover.

Which leads to the uncomfortable conclusion: your research process has to record its own history, because the trial count is data that exists nowhere else. A pipeline that produces winners at a steady rate is not finding more edges than everyone else — it is searching faster, and reporting the maximum. The rejection rate of a sound process is high, and staying comfortable with that is most of the discipline.