How to Evaluate a Backtesting Library

Backtesting libraries are usually chosen by popularity, benchmarked on speed, and trusted by default. That ordering is backwards. What determines whether a library’s output means anything is its fill model, its cost model, and whether you can read the code that produces a number — none of which appear in a feature comparison. This is a set of questions to ask of any engine before you let it tell you a strategy works.

Educational material, not trading advice. Algorithmic crypto trading is high-risk and most retail algo traders lose money.

Question one: when and at what price does an order fill?

This single answer determines more about your results than the strategy does. Ask specifically:

• Which bar fills a signal generated on bar t? If the answer is “bar t”, the engine leaks by default and every result is optimistic. The defensible answer is bar t+1, and you should be able to verify it rather than take it on faith.
• What price within that bar? The open, the close, and the bar’s average are three different assumptions with materially different results, especially on volatile intervals.
• How are stops filled? An engine that fills a stop exactly at its trigger price is modelling a market that always accommodates you. A gap through the level is the normal case in crypto, and the honest model fills at the next available price, which is worse. A library that gets this wrong makes stop-based strategies look like they cap losses precisely.
• Are limit orders filled when the bar merely touches the level? Touching a price is not the same as trading enough volume there for your order to be reached in the queue. Optimistic touch-fills are a common source of a backtest that cannot be reproduced live.
• Is size ever constrained? An engine happy to fill any quantity at the quoted price assumes infinite depth — tolerable for small size on liquid pairs, a fantasy otherwise.

Do not accept documentation as the answer to these. Write a two-bar test case with a known outcome and read the resulting trade log.

Question two: what does it charge you?

Cost handling separates research tools from toys. Look for the ability to specify a per-trade percentage fee, a differentiated maker and taker rate, and a slippage assumption that scales with something — volatility, size, or spread — rather than a flat constant. A library that only supports a fixed commission cannot express the cost structure that actually applies.

Two subtler points. First, whether costs are applied to the notional traded or to the position change: a strategy that rebalances a position from 90% to 100% should pay for the 10% it traded, not the whole position, and engines differ on this. Second, whether funding, borrow, or carry costs can be represented at all — if the strategy holds leveraged or short exposure, those are real recurring costs, and an engine with no place to put them will overstate returns for as long as the position is held. See modelling transaction costs in a backtest.

Question three: can you audit it?

A number you cannot trace is not a result. The properties that make an engine auditable:

• A complete trade log. Every fill with its timestamp, side, size, price, and the cost charged. Without this, you cannot check whether the top few trades carry the entire result — one of the most important diagnostics in how overfitting hides in a trading strategy.
• Readable source for the fill logic. You will need to read it. If the execution model lives in compiled or heavily abstracted code you cannot follow, you are trusting an unexamined assumption at the exact point where assumptions matter most.
• Reproducibility. The same inputs must produce the same outputs, which means any randomness is seeded and any parallelism is deterministic in its result.
• Reconcilable equity. You should be able to reconstruct the reported equity curve from the trade log and the price series by hand. If you can’t, either the log is incomplete or something is happening you don’t know about.

The strongest single test of an engine is to reimplement one simple strategy in twenty lines of pandas and compare. Any material disagreement is worth chasing to its source: you will either find a bug in your code, which is useful, or a modelling assumption in the library you didn’t know about, which is more useful.

Question four: does its data model match your strategy?

Engines differ in what they can express, and the mismatch is expensive to discover late. Check whether it supports multiple assets simultaneously with shared cash, whether position sizing can depend on current equity, whether it can hold several independently managed positions in the same asset, and whether it can consume features you computed elsewhere rather than only indicators it defines.

This is also where the vectorized/event-driven distinction bites. A vectorized engine will be fast enough to sweep parameters but structurally unable to represent order lifecycles and path-dependent exits; an event-driven one can represent them and will be far slower. The full tradeoff is in event-driven vs. vectorized backtests, and the answer is usually to use one of each rather than to pick.

Question five: what does it do about validation and search?

Convenience features here are a mixed blessing. Built-in parameter optimizers and cross-validation splitters save real work, but two cautions apply. Any splitter must respect time ordering — a shuffled k-fold in a trading library is a red flag about the authors’ assumptions, and even correct time-series splits need attention to the embargo between train and test, per how walk-forward validation works.

More importantly, an engine that makes it trivial to test ten thousand parameter combinations makes it trivial to overfit ten thousand times, and it will not tell you how many you tried. Whatever the library offers, keep your own count — the number of configurations evaluated is what determines how much the best result means, per multiple testing and strategy selection.

What matters less than people think

Speed matters only past a threshold. Fast enough to iterate is enough, and optimizing for speed selects for engines that vectorize aggressively, which correlates with weaker fill modelling. Feature count matters little — built-in indicators are trivial to compute, and the engine’s value is in the accounting. Popularity is mainly a proxy for having been read by many people, which is worth something, but a popular engine with an optimistic fill model is still optimistic.

The takeaway

Choose the engine whose execution assumptions you have personally verified and whose costs you can configure to be pessimistic. Then treat its output as one of two independent measurements rather than as the answer: an engine you didn’t write, agreeing with a twenty-line implementation you did, is far stronger evidence than either alone. The library never supplies the rigor — see the Python tooling stack for crypto quant research for where it sits among everything else that does.