Market Data — Integration Guide¶

How to consume normalized market data from feedd. There are three integration tiers, from simplest to most control.

SHM segments¶

Segment names follow the pattern /sorcery-{stack}-{purpose}, where {stack} is master or nightly.

Consumers map all segments read-only. Multiple consumers can map the same segments concurrently — each maintains independent read state.

bookd: pre-built books over UDP¶

bookd is a sidecar that reads the md ring, maintains order books internally, and publishes ready-to-read messages over UDP multicast. Consumers who want pre-built depth and don't need SHM access just join a multicast group and recv().

Every message bookd publishes is fixed-size — no variable-length payloads, no view types, no span math. Just cast the datagram and read.

What bookd publishes:

L3, L4, and SnapshotRef are never published by bookd. The largest datagram is an L2 at 380 bytes (56-byte header + 324-byte payload) — well within the 1472-byte UDP MTU.

Same wire format. Messages use the same common header and payload structs as the md ring. The only differences: L2 depth is always 10/10 and Trade is always a single entry.

Gap recovery is trivial. UDP is lossy — packets may be dropped. But because every L2 is a complete book snapshot (not a delta), the consumer just waits for the next one. No snapshot request, no delta buffering, no reconciliation. Track seq to detect gaps; ignore them for book state since the next L2 overwrites it entirely.

Consumer example (any language):

import socket, struct

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', 5100))

# join multicast group
mreq = struct.pack('4sL', socket.inet_aton('239.1.1.1'), socket.INADDR_ANY)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)

while True:
    data = sock.recv(4096)
    # first 56 bytes = common header
    msg_type = data[46]
    # cast remainder based on msg_type — all payloads are fixed-size

No SHM, no ring, no C++ headers required. Any language that can recv UDP and parse packed structs works.

BookBuilder: in-process convenience¶

BookBuilder is a C++ class in the header library that does the same work as bookd inside the consumer's own process. It wraps a Consumer, drains the md ring, handles L3/L4 application, manages snapshot recovery, and exposes read-only views of the current book state.

Use this when you want convenience without running a separate sidecar, or when you need to combine book state with your own logic in a tight loop.

#include <sorcery/book_builder.h>

sorcery::BookBuilder builder("/sorcery-master-md", "/sorcery-master-metadata");
builder.subscribe({inst_id_1, inst_id_2});

while (running) {
    builder.poll();  // drains ring, applies deltas, manages snapshots

    auto* book = builder.book(inst_id_1);
    // book exposes sorted price levels — iteration, lookup, best bid/ask

    auto* last = builder.last_trade(inst_id_1);
    auto* fund = builder.funding(inst_id_1);
    auto* oi   = builder.open_interest(inst_id_1);
}

Serialization helpers — for consumers who need to republish book state downstream:

// Serialize current book into a wire-format L2 message (same struct as messages.md)
auto l2_msg = book->to_l2(10);    // 10 levels/side

// Serialize last trade as a single-entry Trade message
auto trade_msg = builder.last_trade_msg(inst_id_1);

bookd itself uses BookBuilder internally — the sidecar is a thin wrapper that creates a BookBuilder, polls it, and sends the results as UDP datagrams.

Raw consumer: full control¶

For consumers who need every message type, want to build their own book structures, or require the lowest possible latency. The consumer reads the md ring directly and handles all message types including L3 deltas, L4 order events, and SnapshotRef.

Two sub-options:

Using the sorcery header library¶

The provided C++ headers handle ring framing, gap detection, type casting, metadata seqlock, and price conversion.

#include <sorcery/types.h>     // POD structs, enums
#include <sorcery/ring.h>      // Ring, Consumer, DrainBuffer
#include <sorcery/metadata.h>  // MetadataStore

What the library does for you:

• Opens and validates SHM segments (magic number, schema version)
• Drains messages from the ring into process-local memory (drain-then-process)
• Detects gaps and resets, surfaces them as synthetic frames
• Provides view types for variable-size payloads (L2View, L3View, L4View, TradeView)
• Manages the metadata seqlock, copies instrument/asset structs into local storage
• Converts tick/step integers to real prices and quantities

What you are responsible for:

• Book building and maintenance (applying L3/L4 deltas, managing local book state)
• Gap recovery (requesting snapshots via the Control Plane, buffering and reconciling deltas)
• Sequence tracking per ordering domain if you need gap detection at the application level
• Threading — the library is single-threaded; if multiple threads need market data, you publish from the reader thread to others via your own mechanism

Minimal consumer:

auto ring = sorcery::Ring::open("/sorcery-master-md");
sorcery::Consumer consumer{ring};
sorcery::DrainBuffer buf;

sorcery::MetadataStore meta("/sorcery-master-metadata");
meta.load_venue(sorcery::VENUE_BINANCE);

while (running) {
    for (auto& frame : consumer.drain(buf, 256)) {
        if (frame.is_gap()) {
            // invalidate books, request snapshot
            continue;
        }
        auto* hdr = frame.header();
        auto* body = frame.body();
        // dispatch on hdr->msg_type — see Message Reference
    }
}

Full consumer pattern with dispatch: Message Reference — Consumer pattern.

Raw SHM access¶

If you cannot use the C++ library (different language, custom runtime, kernel bypass), you can map the SHM segments directly and parse the wire format yourself. Everything is POD and little-endian — no serialization framework is involved.

What you need to implement:

1. Ring consumption. shm_open + mmap the market data segment. Read the ring header (magic, version, buffer_size), then poll the committed counter. Parse records using the length-prefixed framing — each record is a u32 size prefix followed by payload bytes, padded to 8-byte alignment. Handle the 0xFFFFFFFF wrap sentinel. Detect overflow by checking committed - local_position <= buffer_size. Full ring layout: Wire protocol — Ring internals.

2. Message parsing. Each record payload is a 56-byte common header followed by the message body. Read msg_type and payload_len from the header, then parse the body according to the message type layouts.

3. Metadata access. shm_open + mmap the metadata segment. Validate the seqlock generation counter (must be even before and after your read, and unchanged). Copy the instrument/asset structs you need into local memory. Full protocol: Metadata — Seqlock protocol.

4. Snapshot reads. When you receive a SnapshotRef message, shm_open + mmap the snapshot segment (if not already mapped). Read len bytes at the given offset, verify the CRC32C checksum, and apply the book state.

5. Gap detection and recovery. Track seq per (venue, msg_type, inst_id) domain. On discontinuity, GAP flag, or ring overflow — invalidate the affected book and request a new snapshot via the Control Plane. Full procedure: Ordering — Recovery.

Key constraints for raw consumers:

• All integers are little-endian
• All structs are packed POD — no padding beyond what the layouts specify
• Prices and quantities are integer tick/step counts, not floats — you must read the instrument metadata to convert
• The ring is SPMC with a single committed counter — load it with acquire ordering
• Never hold pointers into the shared region across iterations — copy data out, then process

Consumer responsibilities¶

What each tier handles:

All tiers must poll the metadata region periodically and convert tick/step values using instrument metadata. See PxQty.

Versioning and compatibility¶

The schema_ver field in the common message header and the version field in the ring and metadata region headers identify the wire format version. Consumers MUST check these on startup and reject unrecognized versions.

Within a major version, changes are additive only:

• New message types may be added (new msg_type values)
• New flags may be added (unused bits in the flags field)
• Reserved fields may be assigned meaning
• Existing struct layouts are never changed

A breaking layout change (field reorder, size change, removed field) increments the major version. Consumers that encounter an unrecognized major version MUST refuse to read the segment rather than silently misparse data.