Raw files: columns and preview

Conventions

One message, several rows: A list inside a message (levels of a book, a batch of trades) becomes one row per element, and all of them keep the message's seq_local. Without it a message with two elements cannot be told from two messages with one.
The venue's names: Every other column keeps the name of the field in the venue's JSON. Where two names differ only by case (Binance e/E, b/B, a/A, u/U; Bybit s/S), the capital one gets a trailing underscore — E_, B_, A_, U_, S_ — because a case-insensitive engine such as DuckDB would otherwise merge the two silently. Chainlink's inner timestamp is payload_timestamp.
Book sides: A level in the source is a price and a size; which side it is on is only said by the array it is in. That name is kept in side exactly as the venue spells it: bids / asks, b / a, BUY / SELL, bid / offer. Trades carry the venue's own side in the same column.
Times: As the venue sends them: milliseconds for Polymarket, Binance, Bybit, OKX and Chainlink; Coinbase's ISO-8601 strings are stored as integer nanoseconds. ts_local is in microseconds; start_ts and end_ts of the active-market rows are in seconds, as in the source.
Numbers: Numbers the venue sends as strings are stored as float64. float32 would not hold what the venues send: it changes 97% of Polymarket prices and 78% of sizes by a rounding error.
Absent is null: A field the message does not have is a null, never a zero. Spot Binance has no event time, so E_ and T are null there.
A zero size is data: size = 0 is how a venue removes a level, so those rows stay. Only Binance's empty trades (p = 0, q = 0, X = NA) are dropped.
Six coins: BTC, ETH, SOL, XRP, DOGE, BNB. Other symbols the collector sees (the dollar rate on Coinbase, hype and zec on Chainlink) are not written. A venue may lack a coin: Coinbase has no BNB, and then there is simply no file.
Order: Rows are in arrival order, almost sorted by ts_local: a handful per million are out of order by a few milliseconds. Row groups do not overlap in time, so a time filter reads only the groups it needs. Sort if you need strict order.

Polymarket

polymarket.<coin>.<market>_<day>.v4.parquet

The up-down market for one coin and one length (5 or 15 minutes): the order book and trades of both outcome tokens, announcements and resolutions, and which market is currently active. One wide table — which columns are filled depends on the message type, named in event_type.

Preview

Loading the preview…

Columns (53, grouped by the kind of message they belong to)

Added by the collector

The first four columns of every raw file. They are the only ones with a _local suffix, so no venue field had to be renamed to make room for them.

Column	Type	Meaning
seq_local	int64	Number of the source message in the collector's log. Rows that came from one message share it — group by it to put the message back together.
ts_local	int64	Receive time at the collector, microseconds since the Unix epoch. The one clock every source shares: sort on it to merge files into one stream.
kind_local	string	Channel tag, with the instrument class: `binance.futures.btc`, `okx.spot.btc`, `polymarket.clob.btc.5m`, `chainlink.rtds`.
source_local	string	Where the message came from: `ws` (the venue's websocket), `rest` (a REST response), `collector` (composed by the collector itself) or `clob-api` (a market outcome that never arrived on the stream and was restored from the venue's API).

Present on every market-feed message

Column	Type	Meaning
event_type	string	Message type: `price_change`, `book`, `best_bid_ask`, `last_trade_price`, `tick_size_change`, `market_resolved` or `new_market`. Null on the active-market and catalogue rows below — tell those apart by `source_local`.
market	string	Market id (the condition id), `0x` followed by 64 hex digits.
asset_id	string	Token id, a decimal uint256 — keep it a string. A market has two tokens, Up and Down, each with a book of its own; the active-market rows say which is which.
timestamp	int64	Event time on the venue, milliseconds.

Order book changes and snapshots (`price_change`, `book`)

price_change gives one row per changed level and always comes as a mirrored pair (the same level seen from the Up and from the Down token). book gives one row per level of the whole book.

Column	Type	Meaning
side	string	`price_change`: `BUY` (bid) or `SELL` (ask). `book`: the name of the array the level came from, `bids` or `asks`. `last_trade_price`: the side of the trade.
price	double	Price of the level, or of the trade: dollars per share, 0.01 to 0.99.
size	double	Size standing at the level after the change — not a delta; `0` means the level was removed. For a trade, the shares traded.
best_bid	double	Top of this token's book after the change (`price_change`, `best_bid_ask`).
best_ask	double	As `best_bid`, the ask side.
hash	string	Always null: the venue's book hash is not stored.
tick_size	double	Current price step, `0.01` or `0.001`. Only in the snapshot a subscription starts with.
last_trade_price	double	Last trade price of the token. Only in that same snapshot.

Top of book, trades, ticks, resolution

Column	Type	Meaning
spread	double	`best_ask − best_bid` as the venue computes it (`best_bid_ask`).
fee_rate_bps	double	Fee rate in basis points on a trade (`last_trade_price`). Always 0 on these markets.
transaction_hash	string	Always null: the on-chain transaction hash of a trade is not stored.
old_tick_size	double	Price step before the change (`tick_size_change`).
new_tick_size	double	Price step after the change.
winning_asset_id	string	The token that settled at 1.00 (`market_resolved`).
winning_outcome	string	`Up` or `Down` (`market_resolved`).
id	string	The venue's numeric market id — not the condition id. `market_resolved`, `new_market` and catalogue rows.
assets_ids	list<string>	Both token ids of the market (`market_resolved`, `new_market`).
tags	list<string>	The market's tags, the same for a whole series.

Market announcement (`new_market`)

An announcement arrives for every series at once, not only the one a file is for.

Column	Type	Meaning
question	string	The market's title.
slug	string	The market's address on the venue's site.
condition_id	string	The market id again (the venue sends it under both names).
outcomes	list<string>	`[Up, Down]`.
clob_token_ids	list<string>	Token ids of the two outcomes.
order_price_min_tick_size	double	The starting price step.
active	bool	Whether the market is open for trading.
taker_base_fee	double	Taker fee; `0` on these markets.
fees_enabled	bool	`false` on these markets.

Active market (rows written by the collector, `source_local = collector`)

Once a second the collector records which market of the series is active; condition_id and outcomes of these rows are in the group above.

Column	Type	Meaning
coin	string	The coin, e.g. `btc`.
interval_min	int16	Length of the market in minutes: `5` or `15`.
start_ts	int64	Start of the market's window, seconds (as in the source).
end_ts	int64	End of the market's window, seconds.
y_token	string	Token id of the Up outcome. The only place that says which `asset_id` is which.
n_token	string	Token id of the Down outcome.

Catalogue rows (`source_local = rest`)

A periodic dump of the venue's market catalogue, restricted to the markets of this file's series. id, question, slug, outcomes and active share their columns with the announcement rows. The list-valued fields arrive as JSON strings and are stored as real lists.

Column	Type	Meaning
conditionId	string	Market id, in the catalogue's camelCase spelling.
outcomePrices	list<string>	Indicative prices of the two outcomes.
clobTokenIds	list<string>	Token ids of the two outcomes.
startDate	string	Start of the market, ISO-8601.
endDate	string	End of the market, ISO-8601.
closed	bool	Whether the market is closed.
orderPriceMinTickSize	double	Price step.
orderMinSize	double	Smallest order size.
liquidityNum	double	Liquidity as the catalogue reports it.
volumeNum	double	Traded volume as the catalogue reports it.
umaResolutionStatus	string	Status of the oracle resolution.
resolutionSource	string	Where the market resolves from.

Left out on purpose

hash and transaction_hash values — Random hex strings that compress to nothing: they were 76% of the Polymarket files and 37% of a whole day (16 GiB instead of 10). The columns are kept, null, so the schema is the one a reader of the venue's feed expects.
description, event_message, fee_schedule of announcements — A condition text that is one template for every market of a coin, a nested copy of fields already stored, and an empty object while fees are off.
Sports-market fields (sports_market_type, line, game_start_time, group_item_title) — Always empty on these markets.
About thirty catalogue fields — Images, texts and counters that appear in no other channel. Seventeen are kept; the catalogue rows are listed above.
Catalogue entries of markets outside this series — A catalogue dump lists a hundred markets; only those of the file's own series are stored.
event_message of market_resolved — Always null on these markets.

Notes

A snapshot book is a list of two books, Up and Down, in one message: both share seq_local and differ in asset_id, so a level is addressed by (seq_local, asset_id, side).
Some outcomes never arrive on the stream (about 3% of markets). The row is added from the venue's API, marked source_local = clob-api.

Binance

binance.<coin>_<day>.v4.parquet

Spot and USD-M futures of one coin in one table, told apart by kind_local (binance.spot.<coin>, binance.futures.<coin>): the best bid and ask, order book deltas, a periodic REST snapshot of the book, and trades.

Preview

Loading the preview…

Columns (22, grouped by the kind of message they belong to)

Added by the collector

The first four columns of every raw file. They are the only ones with a _local suffix, so no venue field had to be renamed to make room for them.

Column	Type	Meaning
seq_local	int64	Number of the source message in the collector's log. Rows that came from one message share it — group by it to put the message back together.
ts_local	int64	Receive time at the collector, microseconds since the Unix epoch. The one clock every source shares: sort on it to merge files into one stream.
kind_local	string	Channel tag, with the instrument class: `binance.futures.btc`, `okx.spot.btc`, `polymarket.clob.btc.5m`, `chainlink.rtds`.
source_local	string	Where the message came from: `ws` (the venue's websocket), `rest` (a REST response), `collector` (composed by the collector itself) or `clob-api` (a market outcome that never arrived on the stream and was restored from the venue's API).

Common

Column	Type	Meaning
e	string	Event type: `bookTicker`, `depthUpdate`, `trade`, or `depth` for a REST snapshot. Null on spot `aggTrade`; spot `bookTicker` carries no event type either, so it is filled in.
s	string	Instrument, e.g. `BTCUSDT`.
E_	int64	Event time sent by the exchange, ms. Null on the spot streams, which carry none.
T	int64	Matching-engine time, ms, usually 1 ms before `E_`. Null on spot.

Best bid and ask (`bookTicker`)

Column	Type	Meaning
b	double	Best bid price (also the price of a bid level in a depth update).
B_	double	Size at the best bid.
a	double	Best ask price.
A_	double	Size at the best ask.
u	int64	Update id; the last update id of a depth frame; for a REST snapshot, `lastUpdateId`.

Book levels (`depthUpdate`, `depth`)

Websocket frames are deltas, one row per changed level. The REST snapshot (e = depth, source_local = rest) is the full book, 1000 levels per side, and is the anchor a reader rebuilds the book from; it has no symbol, so s is filled from the channel name.

Column	Type	Meaning
side	string	The array a level came from: `b` / `a` in a websocket update, `bids` / `asks` in a REST snapshot.
p	double	Price of the level (or of the trade).
q	double	Size standing at the level after the update; `0` removes it. For a trade, the quantity.
U_	int64	First update id in the frame. On spot, `U_` = previous `u` + 1 means no gap.
pu	int64	Last update id of the previous frame (futures only): equal to the previous `u`, the stream was not broken.

Trades (`trade`, `aggTrade`)

Column	Type	Meaning
m	bool	The buyer was the maker: `true` means the aggressor sold.
X	string	Trade type: `MARKET`, `RPI` (retail price improvement) or `NA`.
f	int64	`aggTrade`: first id of the raw trades the exchange merged into this record.
l	int64	`aggTrade`: last id of that range; `l − f + 1` trades.

Left out on purpose

ps, st, M — The futures pair field (always equal to s), an undocumented constant, and a spot field the exchange documents as “ignore”.
Trade ids (t, and a of aggTrade) — Nothing in the file refers to them; a also clashes with the ask price.
Empty trades — Trades with p = 0, q = 0 and X = NA (a few thousand an hour) are not trades.

Notes

Spot has no E_ and no T: the columns are null there rather than zero.
Where only aggTrade exists (spot, later days), f and l are filled and trade rows are absent.

Bybit

bybit.<coin>_<day>.v4.parquet

Spot and linear perpetual of one coin, told apart by kind_local: the order book at two depths and the trades.

Preview

Loading the preview…

Columns (19, grouped by the kind of message they belong to)

Added by the collector

The first four columns of every raw file. They are the only ones with a _local suffix, so no venue field had to be renamed to make room for them.

Column	Type	Meaning
seq_local	int64	Number of the source message in the collector's log. Rows that came from one message share it — group by it to put the message back together.
ts_local	int64	Receive time at the collector, microseconds since the Unix epoch. The one clock every source shares: sort on it to merge files into one stream.
kind_local	string	Channel tag, with the instrument class: `binance.futures.btc`, `okx.spot.btc`, `polymarket.clob.btc.5m`, `chainlink.rtds`.
source_local	string	Where the message came from: `ws` (the venue's websocket), `rest` (a REST response), `collector` (composed by the collector itself) or `clob-api` (a market outcome that never arrived on the stream and was restored from the venue's API).

Order book (`orderbook.1`, `orderbook.50`)

Column	Type	Meaning
topic	string	The channel: `orderbook.<depth>.<symbol>` or `publicTrade.<symbol>`.
depth	int16	Book depth, `1` (top of book) or `50`, taken from the topic so the two streams do not mix. Null on trades.
type	string	`snapshot` replaces the book; `delta` changes single levels. Always `snapshot` on trades.
s	string	Instrument, e.g. `BTCUSDT`.
ts	int64	Time the message was sent, ms.
cts	int64	Matching-engine time, ms, a few ms before `ts`.
side	string	The array a level came from: `b` or `a`.
p	double	Price of the level, or of the trade.
q	double	Size at the level after the update; `0` removes it.
u	int64	Update id; grows by 1 inside a continuous stream.
seq	int64	Matching-engine sequence number. Also on trades.

Trades (`publicTrade`)

Trades arrive in batches, one row per trade; p and seq are shared with the order book group.

Column	Type	Meaning
T	int64	Trade time, ms.
S_	string	Taker side, `Buy` or `Sell` (the exchange's `S`; the underscore keeps it apart from `s`).
v	double	Trade size.
L	string	Tick direction: `PlusTick`, `ZeroPlusTick`, `MinusTick`, `ZeroMinusTick`.

Left out on purpose

Trade i (uuid) — 36 bytes per trade that nothing else refers to.
Trade BT and RPI flags — Block-trade and retail-price-improvement markers; always false here.

OKX

okx.<coin>_<day>.v4.parquet

Spot and perpetual swap of one coin, told apart by instId and kind_local: the order book (400 levels and the top of book) and the trades.

Preview

Loading the preview…

Columns (15, grouped by the kind of message they belong to)

Added by the collector

The first four columns of every raw file. They are the only ones with a _local suffix, so no venue field had to be renamed to make room for them.

Column	Type	Meaning
seq_local	int64	Number of the source message in the collector's log. Rows that came from one message share it — group by it to put the message back together.
ts_local	int64	Receive time at the collector, microseconds since the Unix epoch. The one clock every source shares: sort on it to merge files into one stream.
kind_local	string	Channel tag, with the instrument class: `binance.futures.btc`, `okx.spot.btc`, `polymarket.clob.btc.5m`, `chainlink.rtds`.
source_local	string	Where the message came from: `ws` (the venue's websocket), `rest` (a REST response), `collector` (composed by the collector itself) or `clob-api` (a market outcome that never arrived on the stream and was restored from the venue's API).

Order book (`books`, `bbo-tbt`)

Column	Type	Meaning
channel	string	`books` (400 levels, snapshot then deltas), `bbo-tbt` (top of book on every change) or `trades`.
instId	string	Instrument: `BTC-USDT` (spot) or `BTC-USDT-SWAP`.
action	string	`snapshot` or `update`; `books` only.
ts	int64	Time at the exchange, ms.
side	string	The array a level came from, `bids` or `asks`; for a trade, `buy` or `sell` (the taker).
px	double	Price of the level, or of the trade. The source gives levels as bare arrays; the names come from the exchange's documentation.
sz	double	Size at the level; `0` removes it. For a trade, the size. In contracts on the swap.
numOrders	int32	Number of orders at the level.
seqId	int64	Update number.
prevSeqId	int64	The previous update's `seqId`, `books` only: a match shows the stream was not broken.

Trades (`trades`)

px, sz, side, ts and seqId are shared with the order book group.

Column	Type	Meaning
count	int32	How many trades the exchange merged into this record.

Left out on purpose

The third element of a level — A deprecated field, always "0".
checksum — The book checksum of books: not used.
tradeId, trade source — source would be easily confused with source_local.

Notes

Swap sizes are in contracts: 0.01 BTC, 0.1 ETH, 1 SOL, 100 XRP, 1000 DOGE per contract.

Coinbase

coinbase.<coin>_<day>.v4.parquet

Spot of one coin against USD: the order book, a ticker and the market trades. Coinbase has no BNB, so no coinbase.bnb file exists. All times are ISO-8601 on the wire and are stored here as integer nanoseconds since the epoch.

Preview

Loading the preview…

Columns (21, grouped by the kind of message they belong to)

Added by the collector

The first four columns of every raw file. They are the only ones with a _local suffix, so no venue field had to be renamed to make room for them.

Column	Type	Meaning
seq_local	int64	Number of the source message in the collector's log. Rows that came from one message share it — group by it to put the message back together.
ts_local	int64	Receive time at the collector, microseconds since the Unix epoch. The one clock every source shares: sort on it to merge files into one stream.
kind_local	string	Channel tag, with the instrument class: `binance.futures.btc`, `okx.spot.btc`, `polymarket.clob.btc.5m`, `chainlink.rtds`.
source_local	string	Where the message came from: `ws` (the venue's websocket), `rest` (a REST response), `collector` (composed by the collector itself) or `clob-api` (a market outcome that never arrived on the stream and was restored from the venue's API).

Common

Column	Type	Meaning
channel	string	`l2_data`, `ticker` or `market_trades`.
type	string	`snapshot` when subscribing, `update` afterwards. `l2_data` only.
product_id	string	Instrument, e.g. `BTC-USD`.
timestamp	int64	Time of the frame, ns.
sequence_num	int64	Frame number within the connection, growing by 1.
side	string	Level side `bid` / `offer` (`l2_data`, a real field — Coinbase calls the ask “offer”); trade side `BUY` / `SELL` (`market_trades`).

Order book (`l2_data`)

Column	Type	Meaning
event_time	int64	Time of the level change itself, ns; earlier than `timestamp`.
price_level	double	Price of the level.
new_quantity	double	New size at the level; `0` removes it.

Ticker (`ticker`)

Column	Type	Meaning
price	double	Last trade price (also the price of a `market_trades` record).
best_bid	double	Best bid.
best_ask	double	Best ask.
best_bid_quantity	double	Size at the best bid.
best_ask_quantity	double	Size at the best ask.
volume_24_h	double	Volume over the last 24 hours.

Trades (`market_trades`)

price and side are shared with the groups above; one row per trade, about two per message.

Column	Type	Meaning
size	double	Trade size.
time	int64	Trade time, ns.

Left out on purpose

low_24_h, high_24_h, low_52_w, high_52_w, price_percent_chg_24_h — Derived from the same price over a day or a year.
trade_id — Nothing in the file refers to it.
Heartbeats — A connection-liveness frame; liveness is visible in the data itself.
The USDT-USD rate channel — The price of the dollar, not a coin.

Notes

Some days carry no l2_data at all (the book channel was not collected); type, event_time, price_level and new_quantity are then null for the whole file.

Chainlink

chainlink.<coin>_<day>.v4.parquet

The Chainlink price feed that settles the markets, for one coin: the oracle price and its 30- and 60-second time-weighted averages, one row per update. There is no order book and no tape here.

Preview

Loading the preview…

Columns (12, grouped by the kind of message they belong to)

Added by the collector

The first four columns of every raw file. They are the only ones with a _local suffix, so no venue field had to be renamed to make room for them.

Column	Type	Meaning
seq_local	int64	Number of the source message in the collector's log. Rows that came from one message share it — group by it to put the message back together.
ts_local	int64	Receive time at the collector, microseconds since the Unix epoch. The one clock every source shares: sort on it to merge files into one stream.
kind_local	string	Channel tag, with the instrument class: `binance.futures.btc`, `okx.spot.btc`, `polymarket.clob.btc.5m`, `chainlink.rtds`.
source_local	string	Where the message came from: `ws` (the venue's websocket), `rest` (a REST response), `collector` (composed by the collector itself) or `clob-api` (a market outcome that never arrived on the stream and was restored from the venue's API).

Oracle updates

Column	Type	Meaning
topic	string	Which series: `crypto_prices_chainlink` (the price), `crypto_prices_twap_thirty`, `crypto_prices_twap_sixty`. Later days also carry `crypto_prices`, the spot prices of the exchanges.
type	string	Always `update`.
symbol	string	Instrument in the oracle's spelling: `btc/usd` (`btcusdt` in `crypto_prices`).
timestamp	int64	Time of the frame, ms.
payload_timestamp	int64	Time of the oracle round, a whole second (the inner `timestamp`).
value	double	The value as a number.
window_s	int16	Length of the averaging window, `30` or `60`; null for the plain price.
full_accuracy_value	string	The same value as a decimal string without loss: what the oracle signs.

Left out on purpose

connection_id — An id of the websocket connection.
Other symbols — The feed carries more than our six coins (hype, zec); they are not stored.

Notes

The 5-minute markets settle on twap_thirty before 2026-08-14 and on twap_sixty after it; the 15-minute markets always on twap_sixty.

Raw files: columns and preview

Conventions

Polymarket

Preview

Added by the collector

Present on every market-feed message

Order book changes and snapshots (price_change, book)

Top of book, trades, ticks, resolution

Market announcement (new_market)

Active market (rows written by the collector, source_local = collector)

Catalogue rows (source_local = rest)

Left out on purpose

Notes

Binance

Preview

Added by the collector

Common

Best bid and ask (bookTicker)

Book levels (depthUpdate, depth)

Trades (trade, aggTrade)

Left out on purpose

Notes

Bybit

Preview

Added by the collector

Order book (orderbook.1, orderbook.50)

Trades (publicTrade)

Left out on purpose

OKX

Preview

Added by the collector

Order book (books, bbo-tbt)

Trades (trades)

Left out on purpose

Notes

Coinbase

Preview

Added by the collector

Common

Order book (l2_data)

Ticker (ticker)

Trades (market_trades)

Left out on purpose

Notes

Chainlink

Preview

Added by the collector

Oracle updates

Left out on purpose

Notes

Order book changes and snapshots (`price_change`, `book`)

Market announcement (`new_market`)

Active market (rows written by the collector, `source_local = collector`)

Catalogue rows (`source_local = rest`)

Best bid and ask (`bookTicker`)

Book levels (`depthUpdate`, `depth`)

Trades (`trade`, `aggTrade`)

Order book (`orderbook.1`, `orderbook.50`)

Trades (`publicTrade`)

Order book (`books`, `bbo-tbt`)

Trades (`trades`)

Order book (`l2_data`)

Ticker (`ticker`)

Trades (`market_trades`)