Where Historical Crypto Data Comes From

Historical crypto data comes from four broadly different kinds of source: a venue’s own public API, bulk archive files a venue publishes, third-party aggregators, and data you record yourself. They differ in cost, depth, granularity, and — most importantly — in what they quietly get wrong. Choosing among them is less about which is “best” than about which failure modes you are willing to inherit and detect.

Educational material, not trading advice. Algorithmic crypto trading is high-risk and most retail algo traders lose money.

The four categories

A venue’s public REST endpoints are the most direct source: the venue is the primary record of what traded on it. They are free, generally require no authentication for market data, and return candles, recent trades, and order book snapshots. The constraints are hard limits per request, aggressive rate limiting, and history that often does not extend as far back as you want. Building a long series means paginating thousands of requests.

Bulk archive dumps — flat files a venue publishes for download, typically daily files of trades or candles per pair. When available these are strictly better than paginating an API for the same period: fewer requests, no per-request row cap, and easy resumability. Coverage varies enormously by venue and by how far back the archive goes.

Aggregators sell or give away cross-venue history through one interface. The appeal is breadth and convenience. The catch is that an aggregate price is a construction: someone chose which venues to include, how to weight them, and how to handle a venue being offline. That construction is a modelling decision embedded in your data, and it is rarely documented in enough detail to reproduce. An aggregate price is also not tradeable anywhere, which matters if your strategy assumes it could transact at the price it sees.

Your own recordings are the only source that gives you data nobody else has and coverage that starts the day you begin. Running a collector against public streams and writing to local storage is not difficult, and it is the only way to obtain point-in-time snapshots of things like the tradeable universe — see survivorship bias in crypto datasets. The obvious limitation is that it cannot give you the past.

Granularity, and what to store

Store the finest granularity you can afford, because coarsening is always possible and refining never is. Trades can be aggregated into any bar interval; hourly bars cannot be decomposed into the path within the hour. If a rule ever needs to know whether the high preceded the low, that question is answerable only at a resolution you kept.

The pragmatic middle ground is to keep one-minute bars for a broad universe and full trade data for a small set of pairs you actively study, since trade data for a liquid pair is orders of magnitude larger than its minute bars and the difference compounds across hundreds of pairs.

Whatever you keep, keep it in a compressed columnar format rather than CSV. Columnar layouts let you read one column across years without decompressing everything else, and they preserve types, which CSV does not — a timestamp round-tripped through CSV returns as a string, and that conversion is where timezone bugs enter.

Pagination and rate limits

Fetching long history from an API is a loop, and the loop has three failure modes worth handling from the start.

• Boundary duplication. Most endpoints are inclusive of the requested start, so each request’s first bar repeats the previous request’s last. Deduplicate on the index rather than trusting the source, per data quality checks for crypto price history.
• Silent truncation. A request for a range wider than the per-request row cap returns the cap’s worth of rows, not an error. If your loop advances by the range you asked for rather than by the last row you actually received, you will skip data — and the resulting series looks complete because the gaps are internal.
• Rate limiting. Sustained bulk fetching will trip a limit. Handle it with backoff and respect whatever the venue documents, rather than by retrying immediately. Getting an address throttled or blocked in the middle of a multi-day pull is an avoidable and expensive way to learn this.

The robust pattern is to advance the cursor from the timestamp of the last row returned, persist each page to disk before requesting the next, and make the whole job resumable so an interruption costs one page rather than the run. A fetch that has to complete in one process invocation will, eventually, not complete.

Coverage is not the same as history

A source having data for a date range does not mean the market it describes existed as your backtest assumes. Three distinctions that get conflated:

Listing date versus data start. A pair’s history begins when it started trading on that venue, which differs by venue for the same asset. A cross-venue study that treats the earliest available start as “the beginning” will mix periods where an asset traded in one place with periods where it traded in several.

Symbol identity over time. Tickers get reused, pairs get renamed, and assets undergo redenominations. A single continuous series under one symbol may contain two different things. Any unexplained discontinuity deserves investigation before it is smoothed away.

Downtime versus no trading. A venue being offline and a pair having no trades produce the same absent rows. They mean different things for volatility estimates and for whether the strategy could have acted. Tag the difference where you can establish it.

Choosing per purpose

A rough mapping. Venue APIs for anything where you need to know what actually traded on the venue you would trade on, and for filling recent history. Bulk archives for long backfills when the venue offers them, because they avoid the pagination hazards entirely. Aggregators for breadth in exploratory work, with the understanding that the price is a construction. Own recordings for point-in-time universe snapshots, for order book data, and for anything you want to be certain nobody has retroactively cleaned.

Two habits apply regardless of source. First, fetch once and iterate offline: raw data lands on disk unmodified, and every transformation is a downstream step you can rerun. Second, keep the raw response, not just the parsed frame — when a discrepancy appears months later, the only way to determine whether your parser or the source was wrong is to still have what the source sent.

The unglamorous conclusion

Data acquisition is where the largest share of real quant work goes and where the least interesting problems live: cursors, retries, deduplication, timezone handling, coverage bookkeeping. It is also where the most damaging errors originate, because a flaw in the data flatters every downstream test equally. Validation schemes like walk-forward validation constrain how much you can fool yourself with parameters; they do nothing at all about a series with a skipped week or a price that was quietly reconstructed.

For where this layer sits relative to the rest of a research setup, see the Python tooling stack for crypto quant research.