Data Quality Checks for Crypto Price History

Every result you produce is a function of your data, and crypto price history is dirtier than most people assume: duplicated timestamps, missing intervals, zero-volume filler, prices that violate their own high-low bounds, and occasional values that are simply wrong. The fix is a validation suite that runs on every dataset before it reaches a strategy, fails loudly, and records what it found. This post is the contents of that suite.

Educational material, not trading advice. Algorithmic crypto trading is high-risk and most retail algo traders lose money.

Index integrity first

Before looking at prices, verify the index is a valid time axis. Four properties, each of which silently breaks downstream code:

assert df.index.is_monotonic_increasing        # ordered
assert df.index.is_unique                      # no duplicates
assert df.index.tz is not None                 # timezone-aware
gaps = df.index.to_series().diff().value_counts()   # regular spacing

Non-monotonic indexes arise from concatenating fetched pages out of order, and a rolling window over an unsorted index produces meaningless values with no error. Duplicate timestamps come from overlapping pagination — the last request’s final bar repeated as the next request’s first — and they double-count a period. Naive timestamps invite the timezone mismatches described in resampling OHLCV candles. Irregular spacing is expected for tick data and a bug for bar data; the value_counts() of the diffs should be a single value plus a handful of gaps.

Deduplicate deliberately rather than with a default. Keeping the first or last of a duplicated pair gives different data, and if the two rows differ in their values you have a source inconsistency worth understanding rather than resolving arbitrarily.

Invariant violations

OHLCV rows have internal constraints, and violations mean the row is wrong. Check all of them:

• low <= open <= high and low <= close <= high. A violation means an aggregation bug or a corrupt source. There is no benign explanation.
• low <= high. Sounds redundant; catches columns swapped during a rename or a merge.
• volume >= 0 and prices > 0. A zero or negative price will produce infinities in returns and nonsense in logs.
• No NaN in price columns. Where they exist, understand their origin before filling — see the gap discussion below.

Run these as assertions in the loader, not as an exploratory notebook cell. A check that only runs when you remember to run it is not a check.

Gaps, and what caused them

A missing interval is not one thing, and the right handling depends on which thing it is. Distinguish:

• Coverage boundary. The history simply starts later than you asked. Not a gap — trim the request.
• Venue downtime or collector failure. Real time passed with no record. Forward-filling here invents a flat, calm stretch, which lowers measured volatility and inflates every risk-adjusted metric.
• A genuine no-trade interval. Common in thin pairs. The price legitimately didn’t change because nothing traded, and the correct representation is a forward-filled price with zero volume and a flag saying the bar is untradeable.

The practical move is to keep a synthetic boolean column marking any row your pipeline created rather than received. Every downstream analysis can then choose to exclude synthetic rows, and you never have to reconstruct after the fact which values were real.

Quantify gaps rather than eyeballing them: total missing intervals, longest consecutive run, and the fraction of the sample they represent. A dataset that is 3% synthetic is usable with caveats; one that is 30% synthetic is a different dataset from the one you think you have.

Stale bars and repeated values

Look for runs of identical closes. A long stretch of a repeated price is either a very illiquid pair, a forward-fill someone else applied before you got the data, or a feed that froze. All three matter, and none is visible in summary statistics.

stale = (df["close"].diff() == 0)
runs  = stale.groupby((~stale).cumsum()).sum()
runs.max()          # longest consecutive frozen stretch

Related: bars where open == high == low == close with non-trivial volume. That pattern is possible but unusual, and a large number of them suggests a source that fabricates bars during gaps. If your data has been pre-filled by someone else, your own gap statistics will report a clean dataset while the flat periods quietly damp every volatility estimate.

Outliers you should not delete

A large single-bar move in crypto is often real, and deleting it is the most consequential data decision you can make. Genuine violent moves, liquidation cascades, and thin-book wicks all produce extreme observations that are true records of what the market did. Removing them because they look wrong produces a dataset in which the strategy’s worst case never happens — a self-inflicted survivorship problem, discussed more broadly in survivorship bias in crypto datasets.

Some extremes are, however, artefacts: a decimal-shifted print, a bad tick, a value from a different pair merged in by a symbol collision. Distinguishing them takes evidence, not a threshold:

• Cross-check against another source. If two independent sources show the move, it happened. This is the only decisive test.
• Check whether the move persisted. A price that jumps and immediately returns to the prior level within one bar is more likely a bad tick than a repricing; one that jumps and stays is a repricing.
• Check magnitude against the pair’s own history. A move well outside anything else the pair has done deserves scrutiny, but “outside the usual range” is emphatically not proof of error in an asset class where the usual range is wide.

When you do intervene, tag rather than delete, and record the reason. A dropped row is an undocumented decision that will be invisible to you in six months and will change every result.

Volume and cross-source agreement

Volume is the least standardized field you will handle. Different sources report base volume, quote volume, or a sum across venues, and the same column name may mean different things in two files. If volume feeds a liquidity filter or a cost model, confirm the units — the sanity check is whether volume × typical price is the same order of magnitude as a reported quote volume.

Where two sources cover the same pair and interval, compare them: relative price differences should be tightly concentrated near zero, and a systematic offset means the series are not measuring the same thing.

Making it routine

Put the whole suite behind one function that every dataset passes through on load, returning both the cleaned frame and a report. The report — row count, date range, gap statistics, synthetic fraction, invariant violations, stale runs, flagged outliers — belongs in your research log next to the results it produced, per a reproducible workflow for strategy research.

The reason to invest in this before anything else is a matter of ordering. Validation schemes like walk-forward validation and honest cost models protect you from fitting noise and from optimistic execution assumptions. Neither does anything about a series with duplicated bars, an hour of fabricated calm, or a decimal-shifted print — those flatter every window equally, and a result computed on them is a measurement of your loader, not of the market.