Metadata¶

Metadata is distributed via a shared SHM region written by the metad and read by all other processes. This region contains the full catalog — all assets, all instruments, the string table, the venue registry, and risk metadata. The metad is the sole writer; all other processes (including feedd, orderd, and consumer applications) are read-only.

The core struct definitions for assets and instruments are documented in ID model.

SHM region layout¶

The metad maintains a POSIX shared memory segment (shm_open / mmap) named /sorcery-{stack}-metadata (e.g. /sorcery-master-metadata) containing the complete metadata catalog.

Region header¶

Fixed-size header at the start of the segment (64 bytes, cache-line aligned):

offset  size  type          field
───────────────────────────────────────────────────────
0       4     u32           magic              0x4D455441
4       2     u16           version
6       2                   _pad0
8       8     atomic<u64>   generation         seqlock (odd = write in progress)
16      4     u32           asset_count
20      4     u32           instrument_count
24      4     u32           string_count
28      4     u32           venue_count
32      4     u32           risk_count
36      4     u32           asset_offset       byte offset from segment base
40      4     u32           instr_offset
44      4     u32           string_offset
48      4     u32           venue_offset
52      4     u32           risk_offset
56      8                   _reserved
───────────────────────────────────────────────────────
                            64 bytes total

Data sections¶

The five data sections follow the header. Their positions are given by the offset fields above — consumers MUST use the offsets, not assume a fixed order.

segment base
  ├── [0 .. 64)                        Region header
  ├── [asset_offset  .. +N)            Asset array       N = asset_count × 16
  ├── [instr_offset  .. +N)            Instrument array  N = instrument_count × 56
  ├── [string_offset .. +N)            String table      variable-length entries
  ├── [venue_offset  .. +N)            Venue registry    variable-length entries
  └── [risk_offset   .. +N)            Risk metadata     N = risk_count × 28

Asset array and instrument array are packed fixed-size structs (see ID model). Element i is at offset + i × stride.

String table and venue registry are variable-length. Each entry is self-describing (length-prefixed). Consumers walk the entries sequentially.

Region contents¶

String table¶

Canonical keys are not carried on the hot path. They are delivered in the metadata region as a separate string table, indexed by the u64 identifier:

STRING_TABLE_ENTRY
  id        : u64     asset_id or inst_id
  key_len   : u16
  key_bytes : [u8]    canonical_key, UTF-8

Consumers use this table to resolve between canonical keys and numeric IDs during initialization. The MetadataStore::resolve() method wraps this lookup.

Venue registry¶

The mapping from venue name (as it appears in canonical keys) to the u8 venue byte (as it appears in packed structs):

VENUE_REGISTRY_ENTRY
  venue     : u8      numeric venue ID
  name_len  : u16
  name      : [u8]    "binance", "coinbase", etc.

Risk metadata¶

An optional enrichment layer consumed by quant, risk, and portfolio systems. Neither feedd nor orderd requires these fields to operate.

Asset risk fields — packed fixed-size struct (28 bytes):

offset  size  type   field
─────────────────────────────
0       8     u64    asset_id
8       8     u64    unwrap_to      parent asset_id (0 if root or index)
16      8     u64    risk_root_id   terminal netting bucket
24      1     u8     class          ROOT | WRAPPED | STAKED | BRIDGED | CUSTODIAL | INDEX
25      3            _pad
─────────────────────────────
                     28 bytes total

Asset classes:

Unwrap chain:

The unwrap_to field forms a directed graph from derivative assets to their root. Following the chain terminates at an asset where class == ROOT and risk_root_id points to itself.

syn.binance:eth  (CUSTODIAL, unwrap_to → native.evm:1)
  └── native.evm:1     (ROOT, risk_root_id → self)

erc20.evm:1_<steth>   (STAKED, unwrap_to → native.evm:1)
  └── native.evm:1     (ROOT, risk_root_id → self)

erc20.evm:1_<wbtc>    (WRAPPED, unwrap_to → native.btc)
  └── native.btc        (ROOT, risk_root_id → self)

All assets sharing the same risk_root_id can be netted for exposure and margin calculations.

Seqlock protocol¶

The generation counter uses a seqlock to guarantee consumers never observe a partially written metadata snapshot. The metad bumps generation to an odd value before writing and to an even value after writing. Consumers spin-retry if they observe an odd value or if the counter changed during their copy.

Writer (metad):

1. Store generation = gen + 1  (memory_order_release)    // odd → write in progress
2. Update asset/instrument/string/venue/risk structs
3. Store generation = gen + 2  (memory_order_release)    // even → consistent

Reader (consumer):

1. Load generation (memory_order_acquire)
2. If generation is odd → spin, go to step 1
3. Copy entries of interest into process-local storage
4. Load generation again (memory_order_acquire)
5. If generation changed → discard copy, go to step 1

This guarantees the consumer's local copy is internally consistent. A torn read — where the consumer copies half-old, half-new data from a struct the binary is actively writing — is impossible. Corrupted metadata (wrong tick exponent, wrong settlement asset) could cause catastrophic trading losses; the seqlock eliminates this risk at near-zero cost.

Consumer access pattern¶

Consumers MUST NOT hold pointers into the shared region during the hot path. The recommended access pattern is copy on load with seqlock validation:

1. Map the metadata SHM region (read-only).
2. Read generation (must be even).
3. Scan the catalog and copy the entries of interest into a process-local data structure (e.g. a local hash map keyed by inst_id / asset_id).
4. Read generation again — if it changed, discard and retry from step 2.
5. Unmap the region.
6. All hot-path lookups use the local copy exclusively.

This approach provides three properties:

• No torn reads. The seqlock guarantees the consumer's copy reflects a single consistent snapshot. No partial writes are ever observed.
• No cache pollution. The full catalog may contain thousands of entries the consumer does not use. Copying only the relevant subset keeps the consumer's working set in L1/L2 (56 bytes × 200 instruments ≈ 11 KB).
• No TLB pressure. Repeated lookups into a shared mapping incur TLB misses against a region the consumer does not control. A local copy lives in the consumer's own address space.

Handling updates¶

Metadata changes are delivered exclusively through the SHM region — there is no metadata message type on the data ring. The consumer detects changes by polling the generation counter in the region header:

1. Periodically check if generation has advanced since the last reload().
2. If it has, call reload() to re-map the region and copy changed entries.

Polling is the only mechanism. A fixed interval (e.g., every few seconds) is sufficient — metadata changes are infrequent.

Events that trigger a meta_seq increment¶

• Asset: decimals change, status transition, venue reassignment
• Instrument: fee update, tick/step change, status transition, base/quote/settle reassignment
• Risk tier: class reclassification, unwrap_to change, risk_root_id change

Invariants¶

• meta_seq MUST be monotonically increasing per entity
• class == ROOT → unwrap_to MUST be 0 and risk_root_id MUST be non-zero

Client library¶

The provided header library handles all SHM mechanics. Consumers interact with metadata through a MetadataStore that manages mapping, copying, seqlock validation, and lookup internally.

Loading¶

// Open the metadata SHM region
sorcery::MetadataStore store("/sorcery-master-metadata");

// Selective load by ID
store.load_assets({asset_id_1, asset_id_2, asset_id_3});
store.load_instruments({inst_id_1, inst_id_2});

// Selective load by canonical key
store.load_instruments({"perp.binance:BTCUSDT", "spot.coinbase:BTC-USD"});

// Bulk load all entries for a venue
store.load_venue(sorcery::VENUE_BINANCE);

Each load_* call maps the SHM region, validates the seqlock, copies the requested entries into process-local storage, and unmaps. The consumer never observes the shared region directly.

Lookup¶

// Returns pointer to local copy — safe for hot-path use
const sorcery::Instrument* inst = store.find_instrument(inst_id);
const sorcery::Asset* asset = store.find_asset(asset_id);

// Resolve a canonical key to its u64 ID
std::optional<uint64_t> id = store.resolve("perp.binance:BTCUSDT");

// Convert tick/step integers from wire messages to real values
double price = inst->to_price(px);   // px × (price_tick_mantissa × 10^price_tick_exponent)
double qty   = inst->to_qty(qty);    // qty × (qty_step_mantissa × 10^qty_step_exponent)

All returned pointers reference the consumer's own memory. They remain valid until the next reload() call.

Reloading¶

// Re-map SHM, diff against local state, copy changed entries
// Returns the set of IDs that were updated
std::vector<uint64_t> changed = store.reload();

reload() maps the region, validates the seqlock, compares per-entity meta_seq values against the local copies, and re-copies only entries with a newer meta_seq. Consumers SHOULD call reload() on a periodic timer (e.g. every few seconds) to pick up changes.

Threading¶

The MetadataStore is not thread-safe. Consumers are responsible for synchronization if the store is accessed from multiple threads. The recommended pattern is to call reload() from a single owner thread and publish updates to other threads through the consumer's own concurrency mechanism (e.g. atomic swap of a shared pointer, message passing, or a read-copy-update scheme).