polymarket-canary-tape

Continuous tape from Scribe (Bot E / bot_e_recorder), a single always-on VPS node. Captures co-located CEX trades and Polymarket market-channel WebSocket events over a fixed UTC window for microstructure and lead-lag research.

Where this came from: released alongside polymarket-bot-lab (11 open-source Polymarket trading bot candidates, Apache-2.0) by the team behind OracleMangle, which builds dispute-risk scoring for prediction-market questions. Both artifacts are free; the scorer is the commercial product and is not part of this dataset.

Three tape files, one schema:

The overlap between the canary and vantage-b files (2026-06-03 to 2026-07-01) enables cross-vantage latency and feed-consistency studies: same markets, two independent receive clocks. Vantage-b has richer Polymarket capture in its window (30.9M price_change events, plus market_resolved events absent from the canary schema's event set) and thinner CEX capture (4.3M trades).

Preview samples

The full tapes are zstd-compressed SQLite (the Dataset Viewer cannot render those directly). For browsing, samples/ holds parquet extracts the viewer renders above: the complete markets table plus 100k-row samples of pm_events and cex_trades from the primary canary file. Serious use should decompress the full tapes.

Contents

Row counts above are for the primary canary file. Vantage-b: 33,300,844 pm_events / 4,293,208 cex_trades / 108k+ new_market rows. Early-72h: 531,990 pm_events. Full column definitions: DATA_DICTIONARY.md. Full DDL: schema.sql. PII validation for all three files: VALIDATION.md.

What it is

This is a single-vantage continuous tape: one always-on recorder node subscribed to Polymarket's public market WebSocket channel and to CEX public trade feeds for major crypto symbols. It is not a multi-node or exchange-side archive.

Event types in pm_events are primarily:

• book — full order-book snapshots for an asset token
• price_change — incremental price/size deltas

Timestamps on Polymarket events are local receive times (received_at_ms, UTC milliseconds at the recorder). CEX rows also store exchange trade time (trade_time_ms) alongside local receipt.

How to use

Decompress and open

# Decompress (produces a ~57 GB SQLite file)
zstd -d polymarket-canary-tape-20260513-20260705.db.zst

# Open with the SQLite CLI
sqlite3 polymarket-canary-tape-20260513-20260705.db

Optional: keep the compressed archive and stream-decompress only if your tooling supports it; most research workflows decompress once to disk.

Join Polymarket events to markets

pm_events.condition_id links to markets.condition_id. Asset tokens link via pm_events.asset_id to markets.yes_token_id or markets.no_token_id.

-- Events for a specific market condition
SELECT e.received_at_ms, e.event_type, e.asset_id, e.payload_json
FROM pm_events e
WHERE e.condition_id = '<condition_id>'
ORDER BY e.received_at_ms
LIMIT 100;

-- Join events to the latest known market metadata for that condition
SELECT
  e.received_at_ms,
  e.event_type,
  e.asset_id,
  m.question,
  m.yes_token_id,
  m.no_token_id,
  m.symbol,
  m.duration_minutes
FROM pm_events e
JOIN markets m
  ON m.condition_id = e.condition_id
WHERE e.condition_id = '<condition_id>'
ORDER BY e.received_at_ms, m.scan_at_ms
LIMIT 100;

-- Resolve asset_id to YES/NO side
SELECT
  e.received_at_ms,
  e.event_type,
  e.asset_id,
  CASE
    WHEN e.asset_id = m.yes_token_id THEN 'YES'
    WHEN e.asset_id = m.no_token_id THEN 'NO'
    ELSE 'unknown'
  END AS side,
  m.question
FROM pm_events e
JOIN markets m
  ON m.condition_id = e.condition_id
  AND (e.asset_id = m.yes_token_id OR e.asset_id = m.no_token_id)
WHERE m.symbol = 'BTC'
  AND m.duration_minutes = 15
ORDER BY e.received_at_ms
LIMIT 50;

Book reconstruction (snapshots + deltas)

For a given asset_id, replay in receive-time order:

1. Apply each event_type = 'book' payload as a full replace of the local book.
2. Apply each subsequent event_type = 'price_change' payload as an incremental update until the next book.
3. Use received_at_ms as the timeline; do not assume exchange-side sequencing beyond what the payload encodes.

-- Ordered stream for one token (reconstruct offline)
SELECT received_at_ms, event_type, payload_json
FROM pm_events
WHERE asset_id = '<token_id>'
  AND event_type IN ('book', 'price_change')
ORDER BY received_at_ms;

Index ix_pm_events_asset_type_time on (asset_id, event_type, received_at_ms) is intended for this pattern.

CEX trades near a Polymarket window

-- BTCUSDT trades in a local-receive window
SELECT trade_time_ms, received_at_ms, price, size, is_buyer_maker
FROM cex_trades
WHERE symbol = 'BTCUSDT'
  AND received_at_ms BETWEEN 1747000000000 AND 1747003600000
ORDER BY trade_time_ms;

Liveness

SELECT source, COUNT(*) AS n, MIN(emitted_at_ms), MAX(emitted_at_ms)
FROM heartbeats
GROUP BY source;

SELECT COUNT(*) FROM gaps;  -- 0 in this release

Methodology

• Single node. One always-on VPS recorder; all streams share the same local clock for received_at_ms / emitted_at_ms.
• Public channels only. Polymarket market channel (anonymous by design). No user channel, no authenticated APIs, no order placement logs.
• Heartbeats. Table heartbeats records periodic liveness from sources pm, cex, and discovery, with optional last_message_age_sec.
• Gaps. Table gaps is the outage log. This release has 0 rows — no recorded outages in the capture window.
• Market metadata. markets rows are discovery/scan snapshots (scan_at_ms), not a single static registry. The same condition_id may appear more than once as scans refresh prices and volume.
• Schema version. schema_version.version = 1 for this tape format.

Limitations

1. Single vantage point. Latency, packet loss, and reconnect behavior are those of one recorder, not an exchange-side or multi-region consensus feed.
2. Market channel only. No user identities, wallet addresses, fills attributed to accounts, or private order state. Anonymous public book traffic only.
3. Local receive timestamps. received_at_ms is when the recorder accepted the message. Network delay and queueing are embedded; CEX trade_time_ms is the exchange-reported trade time when present.
4. Not a complete market universe. Only subscriptions the node held during the window appear in pm_events. markets reflects discovery scans, not every Polymarket market ever listed.
5. Empty gaps is not a formal SLA. It means the recorder did not log outages under its gap rules; it does not prove perfect upstream availability.
6. Payloads are raw JSON. Field layouts inside payload_json / raw_json follow upstream APIs as of capture time and may require defensive parsing.

Intended uses

• Prediction-market microstructure research (book dynamics, snapshot vs delta cadence)
• Prediction-market vs CEX lead-lag and co-movement studies
• Offline backtest replay of public book state against CEX tape
• Method development for gap detection, liveness scoring, and multi-stream alignment

Not intended for: trading as live market data, reconstructing private user activity, or claiming multi-region ground truth.

Licence and citation

This dataset is licensed under Creative Commons Attribution 4.0 International (CC-BY-4.0). See LICENSE.

Please cite the dataset name, window, and file when publishing:

polymarket-canary-tape (2026-05-13 to 2026-07-05 UTC).
File: polymarket-canary-tape-20260513-20260705.db.zst.
Licence: CC-BY-4.0.

Provenance

Recorded by Scribe (Bot E), the market-recorder lane of the retired longshot-research project. Companion code (strategies and tooling, not this tape) lives in polymarket-bot-lab (link TBD).

The same project also used a separate closed scorer product, Oraclemangle, which is unrelated to and absent from this data. This release contains only the public canary tape described above.

Files in this release