Wire Protocol¶

Both feedd and orderd publish messages over shared memory ring buffers. The transport is lock-free, zero-copy on the producer side, and requires no kernel involvement on the data path.

SHM naming convention¶

All POSIX shared memory segments follow the pattern /sorcery-{stack}-{purpose}, where {stack} is master (stable) or nightly (latest). The stack name is set in configuration. Each product documents its specific segments in its own integration guide.

Client library¶

The provided header library handles all SHM mechanics. Consumers interact with the ring through a small, opaque API — no direct access to shared memory counters, offsets, or buffer layout is required.

Opening a ring¶

#include <sorcery/ring.h>

// Maps the POSIX shared memory segment (read-only for consumers).
// Validates magic number and schema version. Throws on mismatch.
auto ring = sorcery::Ring::open("/sorcery-master-md");

Consumer API¶

sorcery::Consumer consumer{ring};

while (running) {
    auto msg = consumer.poll();
    if (msg.empty()) continue;

    if (consumer.has_gap()) {
        // Producer lapped this consumer — data was lost.
        // Product-specific recovery (see integration guide).
        handle_gap();
        consumer.reset();
        continue;
    }

    dispatch(msg);  // msg is a span<const std::byte> over one message (common header + body)
}

Method	Returns	Purpose
`poll()`	`std::span<const std::byte>`	Next message bytes (common header + body), or empty if no new data. Handles counter caching, sentinel detection, wrap-around, and alignment internally.
`has_gap()`	`bool`	`true` if the producer has overwritten data this consumer has not yet read. The consumer MUST NOT call `poll()` again until `reset()` is called.
`reset()`	`void`	Re-synchronizes the consumer to the current ring position. All missed messages are discarded.

This is the low-level API. Each product's integration guide documents the recommended higher-level consumption pattern built on top of it.

Producer API¶

sorcery::Producer producer{ring};

// Reserve space, write directly into ring memory, commit.
auto buf = producer.get_buffer(msg_size);
encode_message(buf);
producer.flush();

Method	Returns	Purpose
`get_buffer(size)`	`std::span<std::byte>`	Writable region in the ring for `size` bytes. Handles wrap-around and sentinel insertion internally.
`flush()`	`void`	Atomically publishes all unflushed messages. Consumers see the entire batch or nothing.

flush() may be deferred across multiple get_buffer / write cycles to batch messages. A single flush() at the end of a burst reduces cache-line traffic.

Message format¶

Each message returned by poll() is a plain byte span containing the common message header followed by the message-type-specific body. The framing (length prefix, alignment padding) is stripped by the library — the consumer receives message bytes only (header + body), not ring framing.

All messages MUST be Plain Old Data (POD) — no pointers, no virtual functions, no non-trivial constructors or destructors. The shared memory segment may be mapped at different virtual addresses in each process; embedded pointers would be meaningless in any process other than the writer.

Timestamps¶

Three timestamp domains appear in every message header:

Field	Clock source	Meaning
`exch_ts`	Exchange	Timestamp assigned by the venue (when available)
`rx_ts`	Local host	Time the raw data was received from the network
`pub_ts`	Local host	Time the normalized message was published to the ring

All timestamps are nanoseconds since Unix epoch (uint64_t).

Do not use timestamps for ordering

Timestamps reflect wall-clock observations and are subject to clock skew, NTP adjustments, and processing jitter. Message ordering MUST be determined exclusively by sequence numbers within the appropriate ordering domain. Each product defines its own ordering semantics — see the product-specific documentation.

Ring internals¶

This section documents the internal mechanics of the SHM ring buffer. It is reference material for developers who need to understand the transport layer behavior, debug integration issues, or implement a custom consumer outside the provided library.

Memory layout¶

Each ring is a POSIX shared memory segment (shm_open / mmap) containing a fixed-size circular buffer. The producing binary is the sole writer. Consumer processes map the segment read-only and maintain independent read positions.

offset  size   field
──────────────────────────────────────────────────────────
0       8      magic              0x4D475348514D4B54
8       4      version            schema version (u32)
12      4      buffer_size        usable buffer capacity in bytes (u32)
──────────────────────────────────────────────────────────
        ← cache line boundary (64 bytes) →
64      8      committed          std::atomic<uint64_t>
──────────────────────────────────────────────────────────
        ← cache line boundary →
128     N      buffer[]           circular message buffer
──────────────────────────────────────────────────────────

magic: Identifies the segment as a valid ring. Consumers MUST verify before reading.
version: Schema version. Consumers MUST reject segments with an unrecognized version.
buffer_size: The capacity of buffer[] in bytes.
committed: Monotonically increasing logical byte cursor. All records strictly before this cursor are fully written and safe to read.
buffer: The message data region. Fixed size, allocated at segment creation.

Cache-line discipline¶

The committed counter occupies its own 64-byte cache line via alignas(64). Without this, producer stores to the counter would invalidate cache lines containing the header or buffer data, causing coherence traffic on every message.

struct Ring {
    alignas(64) Header      header;
    alignas(64) std::atomic<uint64_t> committed{0};
    alignas(64) std::byte   buffer[];
};

Record framing¶

Each record in the buffer is a length-prefixed POD message:

┌────────────┬──────────────────────────────┐
│  size (u32) │  payload (size bytes)         │
└────────────┴──────────────────────────────┘
              ← aligned to 8-byte boundary →

size: 4 bytes, little-endian. Byte length of the payload (excluding the size prefix itself).
Alignment: Each record is padded to the next 8-byte boundary. This ensures the size field of the next record is always naturally aligned.
Sentinel: The reserved size value 0xFFFFFFFF is a padding sentinel indicating wrap-around — not a valid message. See Wrap-around.

Commit protocol¶

The producer uses a single logical cursor (committed) to publish messages. Data is written first, then the logical cursor is advanced with a release store to make it visible.

1. Compute `record_size = sizeof(u32) + align_up(msg_size, 8)`
2. Set `logical = local_committed` and `physical = logical % buffer_size`
3. Check remaining physical space (see Wrap-around)
4. `memcpy` size prefix into `buffer[physical]`
5. `memcpy` payload into `buffer[physical + 4]`
6. Advance `logical += record_size`
7. Store `logical` → `committed` (`memory_order_release`)

The memcpy operations (steps 3–4) are ordinary stores — they are not visible to consumers until the release store in step 6.

Why not two counters? A dual-counter design (separate write-cursor and read-cursor) signals write-in-progress and write-complete independently. This introduces a formal data race window, requires two atomic stores per message, doubles the cache-line footprint, and complicates memory ordering — all for no benefit in a single-producer topology. A single counter eliminates these issues.

Operation	Order	Purpose
Producer stores `committed`	`release`	Makes all preceding writes visible to consumers
Consumer loads `committed`	`acquire`	Synchronizes with producer's release — all data before this offset is safe to read

No compare-and-swap (CAS) is required. The single-writer constraint eliminates all contention on the write path.

Wrap-around¶

When the next message does not fit in the remaining physical space, the producer writes a 4-byte padding sentinel (0xFFFFFFFF) at the current physical position, advances the logical cursor to the next buffer lap, writes the message at physical offset 0, and commits everything with a single atomic store.

Producer wrap sequence:

1. `physical = logical % buffer_size`
2. If `physical + record_size > buffer_size`:
   a. Write `0xFFFFFFFF` at `buffer[physical]`
   b. Advance `logical += (buffer_size - physical)` (next lap)
   c. Set `physical = 0`
3. Write record at `buffer[physical]`
4. Advance `logical += record_size`
5. Store `logical` → `committed` (`memory_order_release`)

The sentinel and the wrapped message are committed atomically — consumers never observe a state where the sentinel is visible but the wrapped message is not.

Consumer wrap handling:

When a consumer reads a size prefix of 0xFFFFFFFF, it advances its local logical cursor to the next buffer lap and reads the next message from physical offset 0. Each consumer encounters the sentinel independently at its own read position. No coordination is required.

size_t physical = local_cursor % buffer_size;
uint32_t size = read_u32(buffer, physical);
if (size == PADDING_SENTINEL) {
    local_cursor += (buffer_size - physical);
    physical = 0;
    size = read_u32(buffer, physical);
}

A single message MUST NOT exceed buffer_size - 4 bytes.

Consumer read sequence¶

1. Load `committed` (`memory_order_acquire`)
2. If `local_cursor == committed` → no new data, return
3. Compute `physical = local_cursor % buffer_size`
4. Read size prefix at `physical`
5. If size is `0xFFFFFFFF`:
   a. Advance `local_cursor += (buffer_size - physical)` (next lap)
   b. Go to step 3
6. Read payload
7. Advance `local_cursor += align_up(sizeof(u32) + size, 8)`

Caching: Consumers SHOULD cache the last observed committed value locally. Only reload from shared memory when the cached value is exhausted:

if (local_cursor == cached_committed) {
    cached_committed = ring->committed.load(std::memory_order_acquire);
}
if (local_cursor == cached_committed) return;  // no data

Overflow detection¶

If the producer wraps and overtakes a slow consumer, the consumer's read position references overwritten data. Consumers MUST check:

committed - local_cursor ≤ buffer_size

If violated, the consumer has been lapped. It MUST re-synchronize to the current committed offset and discard missed messages. This is the gap event surfaced by Consumer::has_gap().