Market Data — Message Reference¶

All market data messages share a common header. The consumer reads the header, dispatches on msg_type, and casts the remaining bytes to the appropriate payload struct.

The provided header library (sorcery/types.h) contains all POD struct definitions and enums. For ring mechanics (poll, has_gap, reset), see Wire protocol — Client library.

Consumer pattern¶

Pin one thread per ring. Each iteration drains a batch of frames into process-local memory, then processes them. The drain phase is fast (memcpy only) — once frames are copied out, the ring position advances and the producer has room. The consumer then processes from its own memory at whatever speed it needs.

#include <sorcery/types.h>     // Header, L1Msg, PxQty, L2View, L3View, L4View, TradeView, ...
#include <sorcery/ring.h>      // Ring, Consumer, DrainBuffer
#include <sorcery/metadata.h>  // MetadataStore — instrument lookup + price conversion

auto ring = sorcery::Ring::open("/sorcery-master-md");
sorcery::Consumer consumer{ring};
sorcery::DrainBuffer buf;

while (running) {
    // Phase 1: drain — copy up to K frames out of the ring.
    // Fast (memcpy only). Ring position advances immediately.
    for (auto& frame : consumer.drain(buf, K)) {

        if (frame.is_gap()) {
            invalidate_books();
            request_snapshot();
            continue;
        }

        // Phase 2: dispatch on msg_type, cast payload to the appropriate struct.
        auto* hdr  = frame.header();     // sorcery::Header*
        auto* body = frame.body();       // raw bytes after header

        switch (hdr->msg_type) {
            case sorcery::L1:
                on_l1(*hdr, *reinterpret_cast<const sorcery::L1Msg*>(body));
                break;
            case sorcery::L3:
                on_l3(*hdr, sorcery::L3View{body});   // view provides bid_updates() / ask_updates() spans
                break;
            case sorcery::TRADE:
                on_trade(*hdr, sorcery::TradeView{body});
                break;
            case sorcery::SNAPSHOT_REF:
                on_snap(*hdr, *reinterpret_cast<const sorcery::SnapshotRef*>(body));
                break;
            // ... remaining types
        }
    }
}

drain() wraps the low-level poll() / has_gap() / reset() API (see Wire protocol). It copies up to K frames into the drain buffer and checks for overflow internally. If the producer lapped the consumer, drain() calls reset() and inserts a synthetic gap frame — the consumer sees it as frame.is_gap() and handles recovery without touching the low-level API directly.

Fixed-size payloads (L1, Funding, Liquidation, OpenInterest, SnapshotRef, Status) can be cast directly. Variable-size payloads (L2, L3, L4, Trade) are wrapped by view types (L2View, L3View, L4View, TradeView) that compute array offsets and return typed std::spans — no manual pointer math.

Price and quantity conversion¶

Prices and quantities on the wire are integers in tick/step units. Convert to real values using the instrument metadata (see Metadata):

sorcery::MetadataStore store("/sorcery-master-metadata");
store.load_venue(hdr->venue);  // map SHM, copy to local memory

auto* inst = store.find_instrument(hdr->inst_id);
double price = inst->to_price(l1.bid_px);  // ticks → real price
double qty   = inst->to_qty(l1.bid_qty);   // steps → real quantity

Why drain-then-process¶

Processing directly from ring memory is dangerous — if a handler takes too long, the producer laps the consumer and overwrites the data being read. Copying frames into local memory first means the ring position advances immediately during the drain phase, giving the producer room. The consumer then processes from its own memory with no time pressure.

Rings are sized small — typically hundreds of KB to low single-digit MB — so the working set stays in L1/L2 cache. This means headroom is measured in thousands of messages, not millions. The drain phase must keep up with the producer, which is straightforward since it does nothing but memcpy.

Backpressure¶

The producer never blocks. If the consumer falls behind the ring, the producer overwrites old data. drain() detects this, re-synchronizes the consumer, and inserts a synthetic gap frame. On gap:

Mark all books sourced from this ring as INVALID
Request a new snapshot via the Control Plane
Resume draining — apply the snapshot when it arrives, then continue with deltas

There is no retry, no backfill, no retransmit from the ring. Detect loss, invalidate, resync.

Enums¶

msg_type (u8)¶

Value	Name	Payload size
1	`L1`	32 bytes (fixed)
2	`L2`	variable
3	`L3`	variable
4	`L4`	variable
5	`SNAPSHOT_REF`	40 bytes (fixed)
6	`TRADE`	variable
7	`FUNDING`	40 bytes (fixed)
8	`LIQUIDATION`	24 bytes (fixed)
9	`OI`	16 bytes (fixed)
10	`STATUS`	32 bytes (fixed)

side (u8)¶

Value	Name
1	`BID`
2	`ASK`

Also used for trade aggressor: BID = buyer aggressor, ASK = seller aggressor, 0 = unknown.

event_type (u8)¶

Used in L4 order events.

Value	Name	Meaning
1	`ADD`	New order placed on book
2	`CANCEL`	Order removed (cancel, full fill, rejection)
3	`MODIFY`	In-place amendment (may preserve queue priority)
4	`FILL`	Partial fill — qty field is the filled amount

flags (u16, bitfield)¶

Bit	Name	Meaning
0	`GAP`	Upstream seq discontinuity detected by the adapter
1	`RESET`	Epoch changed — all prior ordering is invalidated
2	`DROP`	Records were dropped due to ring overrun
3	`DERIVED`	Data synthesized by the adapter (e.g., L1 derived from L2)
4	`SNAPSHOT`	First update after a snapshot was applied

Common header¶

Every message begins with this 56-byte header.

Sequence numbers are monotone within (venue, msg_type, inst_id). A seq discontinuity means data was lost — the consumer MUST treat the affected book as invalid until resync.

offset  size  type   field
────────────────────────────────────────────────────────────
0       8     u64    inst_id           instrument ID (0 for Status)
8       8     u64    exch_ts           nanos since epoch; 0 if venue does not provide
16      8     u64    rx_ts             nanos since epoch; adapter socket-read time
24      8     u64    pub_ts            nanos since epoch; ring publish time
32      8     u64    seq               monotone within (venue, msg_type, inst_id)
40      4     u32    epoch             incarnation counter; increments on reconnect/reset
44      2     u16    schema_ver        wire format version
46      1     u8     msg_type          see enum
47      1     u8     venue             venue ID from metadata region
48      2     u16    flags             see bitfield
50      2     u16    payload_len       byte length of payload after this header
52      4     u32    _reserved
────────────────────────────────────────────────────────────
                     56 bytes

payload_len is the byte length of the message-specific body that follows the header. The total frame size (as seen in ring framing) is 56 + payload_len.

Epoch changes invalidate ordering

When epoch changes for a given (venue, msg_type, inst_id) triple, the consumer MUST discard all prior seq state for that triple. The RESET flag is also set on the first message of a new epoch.

Shared types¶

PxQty (16 bytes)¶

Used in L1, L2, and L3 payloads.

offset  size  type   field
────────────────────────────────────────
0       8     i64    px       tick units
8       8     i64    qty      step units

Tick and step encoding. Prices and quantities are integers representing counts of the instrument's minimum increment. Convert to real values using the instrument metadata from the metadata region:

real_price    = px  × price_tick    (where price_tick = mantissa × 10^exponent)
real_quantity = qty × qty_step      (where qty_step  = mantissa × 10^exponent)

Values MUST be non-negative. qty = 0 has type-specific meaning (see L3).

L1: Top of Book¶

Best bid and ask. One message per instrument update.

Payload: 32 bytes. Total frame: 88 bytes.

offset  size  type   field
────────────────────────────────────────
0       8     i64    bid_px       ticks
8       8     i64    bid_qty      steps
16      8     i64    ask_px       ticks
24      8     i64    ask_qty      steps

When the adapter synthesizes L1 from an L2 feed (e.g., Hyperliquid), the DERIVED flag is set.

L2: Top N Levels¶

Periodic depth snapshot from the exchange — a consistent slice of the top N price levels on each side. Levels are ordered best to worst.

Payload: variable. Minimum 4 bytes.

offset  size  type              field
────────────────────────────────────────
0       1     u8                n_bids        levels on bid side
1       1     u8                n_asks        levels on ask side
2       2     u16               _pad
4       ...   PxQty[n_bids]     bids          best → worst
...     ...   PxQty[n_asks]     asks          best → worst

Payload size: 4 + (n_bids + n_asks) × 16 bytes.

Typical sizes: 5 levels/side = 164 bytes payload; 10 levels/side = 324 bytes payload.

L3: Level Delta¶

A batch of price-level changes. One frame is one atomic update — apply all entries, then the book is consistent. Do not read the book mid-application.

Payload: variable. Minimum 4 bytes.

offset  size  type              field
────────────────────────────────────────
0       1     u8                n_bid_updates
1       1     u8                n_ask_updates
2       2     u16               _pad
4       ...   PxQty[n_bid_updates]    bid updates
...     ...   PxQty[n_ask_updates]    ask updates

Payload size: 4 + (n_bid_updates + n_ask_updates) × 16 bytes.

Quantities are absolute, not incremental

Each qty value is the new total quantity at that price level, not a delta from the previous value. qty = 0 means remove the level entirely. Applying the same update twice is idempotent — this property is load-bearing for snapshot reconciliation.

Book building from L3¶

The consumer builds and maintains their own order book. The procedure:

Request a snapshot via the Control Plane
Buffer incoming L3 frames while waiting
Receive SnapshotRef → read the full book from the snapshot SHM region
Discard any buffered L3 where seq <= snap_seq
Apply remaining buffered L3 frames in seq order
Continue applying L3 frames as they arrive
On gap (seq discontinuity or GAP flag) → mark book INVALID, go to step 1

The adapter handles all venue-specific sequencing (e.g., Binance U/u reconciliation) internally. The consumer never needs venue-specific logic.

L4: Order Delta¶

Order-by-order events. One frame is one atomic batch — apply all events before reading the book.

Payload: variable. Minimum 4 bytes.

offset  size  type                  field
────────────────────────────────────────
0       2     u16                   n_events
2       2     u16                   _pad
4       ...   OrderEvent[n_events]

Payload size: 4 + n_events × 32 bytes.

OrderEvent (32 bytes)¶

offset  size  type   field
────────────────────────────────────────
0       8     u64    order_id      deterministic ID assigned by the adapter
8       8     i64    px            ticks
16      8     i64    qty           steps (see below)
24      1     u8     event_type    ADD / CANCEL / MODIFY / FILL
25      1     u8     side          BID / ASK
26      6            _pad

qty interpretation by event_type:

event_type	qty meaning
`ADD`	Order size
`CANCEL`	Remaining size at cancellation (0 if fully cancelled)
`MODIFY`	New order size after amendment
`FILL`	Filled quantity (not remaining — the amount that traded)

order_id contract. For venues with numeric order IDs (e.g., Hyperliquid oid): passed through directly. For venues with string or UUID order IDs: the adapter applies xxHash64 to produce a deterministic u64. The same venue string always maps to the same order_id within an epoch.

SnapshotRef¶

A pointer to bulk book data in a separate SHM region. Snapshots are never inlined on the hot ring — they would be megabytes and shred every consumer's cache.

Payload: 40 bytes. Total frame: 96 bytes.

offset  size  type   field
────────────────────────────────────────
0       8     u64    seg_id         segment in snapshot SHM region
8       8     u64    offset         byte offset within segment
16      8     u64    snap_seq       seq at snapshot creation
24      4     u32    len            decompressed payload size in bytes
28      4     u32    checksum       CRC32C of the payload
32      1     u8     snap_type      1 = L2_BOOK, 2 = L4_ORDERS
33      1     u8     _pad
34      2     u16    depth          book depth; 0 = full depth
36      4     u32    _reserved

Snapshots are always stored decompressed in the snapshot SHM region. The adapter handles decompression before writing.

snap_seq maps to the adapter's sequence space — the consumer uses it to determine which buffered deltas to discard (see L3 book building).

Trade¶

One or more trade executions. Usually n_trades = 1, but venues that send trade bursts (e.g., Binance aggregated trades) are packed into a single frame.

Payload: variable. Minimum 4 bytes.

offset  size  type                  field
────────────────────────────────────────
0       2     u16                   n_trades
2       2     u16                   _pad
4       ...   TradeEntry[n_trades]

Payload size: 4 + n_trades × 32 bytes.

TradeEntry (32 bytes)¶

offset  size  type   field
────────────────────────────────────────
0       8     i64    px            ticks
8       8     i64    qty           steps
16      8     u64    trade_id      venue trade ID; 0 if absent
24      1     u8     aggressor     BID=1 (buyer), ASK=2 (seller), 0=unknown
25      1     u8     flags         bit 0: is_block, bit 1: is_liquidation
26      6            _pad

For venues with string trade IDs (e.g., Bybit): the adapter hashes to u64 using xxHash64.

Funding¶

Funding rate update. Includes optional mark and index prices where the venue provides them.

Payload: 40 bytes. Total frame: 96 bytes.

offset  size  type   field
────────────────────────────────────────
0       8     i64    funding_rate      rate × 10^8 (0.01% = 10,000)
8       8     u64    funding_ts        nanos; when this rate applies
16      8     u64    next_funding_ts   nanos; next funding event
24      8     i64    mark_px           ticks; 0 if absent
32      8     i64    index_px          ticks; 0 if absent

Liquidation¶

Forced liquidation event from the venue.

Payload: 24 bytes. Total frame: 80 bytes.

offset  size  type   field
────────────────────────────────────────
0       8     i64    px       ticks
8       8     i64    qty      steps
16      1     u8     side     BID / ASK
17      7            _pad

OpenInterest¶

Total open interest for an instrument.

Payload: 16 bytes. Total frame: 72 bytes.

offset  size  type   field
────────────────────────────────────────
0       8     i64    open_interest    qty_step units
8       8     i64    notional         quote currency ticks; 0 if absent

Status¶

Connection health and heartbeat. Emitted periodically by the adapter. inst_id is 0 in the header — this message is per-venue, not per-instrument.

Payload: 32 bytes. Total frame: 88 bytes.

offset  size  type   field
────────────────────────────────────────
0       8     u64    last_rx_age_ns      nanos since last exchange message
8       4     u32    reconnect_count
12      4     u32    gap_count           total gaps detected by adapter
16      4     u32    drop_count          total drops (ring overruns)
20      4     u32    active_instruments  instruments currently streaming
24      4     u32    queued_snapshots    snapshots in flight
28      1     u8     conn_state          1=CONNECTED, 2=DISCONNECTED, 3=RECONNECTING
29      3            _pad

If last_rx_age_ns exceeds the expected heartbeat interval for the venue, the consumer should treat the connection as stale.

Venue support matrix¶

Not every venue emits every message type. A dash means the adapter does not produce this type for that venue.

Type	Binance	Bybit	Coinbase	Hyperliquid
L1	yes	yes	yes	yes
L2	yes	yes	yes	yes
L3	yes	yes	yes	—
L4	—	—	—	yes
Trade	yes	yes	yes	yes
Funding	yes	yes	—	yes
Liquidation	yes	yes	—	yes
OI	yes	yes	—	yes