Manual · Storage

Strand Storage

SapixDB persists data as a sequence of append-only binary segment files backed by a write-ahead log. This page explains the on-disk layout, how segments are sealed, how the WAL enables crash recovery, and how chain integrity is verified.

On-disk directory layout

Three root directories hold all persistent data. Each is configurable via an environment variable.

$SAPIX_STRAND_DIR/          # default: /data/strand
  <agent_id>/
    segment_00000.bin       ← sealed, immutable (~4 MB)
    segment_00001.bin       ← sealed, immutable
    segment_00002.bin       ← active (current writes land here)
    wal.bin                 ← write-ahead log (active)
    chain.bin               ← chain head pointer + metadata

$SAPIX_GRAPH_DIR/           # default: /data/graph
  edges/                    ← directional typed edges between agents
  meta/                     ← graph metadata (FTS indexes, etc.)

$SAPIX_BLOB_DIR/            # default: /data/blobs
  <sha256-hash>             ← content-addressed blob storage

Segment files

Each agent's strand is stored as a sequence of numbered segment files (segment_00000.bin, segment_00001.bin, …). Segments are pure append-only binary files — writes only ever extend the tail of the active segment.

Active vs. sealed segments

At any moment there is exactly one active segment per agent — the one currently receiving new records. When the active segment reaches approximately 4 MB it is sealed: its content is finalised, a new segment file is created, and all subsequent writes go to the new file.

Write-ahead log (WAL)

Before any record write is acknowledged to the caller, it is first appended to wal.bin in the same agent directory. Only after the WAL entry is durable does the API return a 200 OK.

Crash recovery

On startup the agent checks whether wal.bin contains uncommitted entries. If it does, it replays them into the active segment before accepting new writes. This guarantees that no acknowledged write is ever lost, even if the process crashes between the WAL write and the segment write.

1. Agent starts
2. Open wal.bin for agent_id
3. If wal.bin has uncommitted entries → replay into active segment
4. Clear / truncate wal.bin
5. Begin accepting API requests

WAL entry contents

Chain integrity

Every record stores a cryptographic hash that chains it to its predecessor. This makes tampering with any past record detectable without replaying the entire data set.

content_hash = BLAKE3(
  parent_hash    ‖   // hash of the previous record (zeros for first)
  timestamp_hlc  ‖   // 8-byte HLC timestamp
  flags          ‖   // 1-byte flags
  payload_msgpack    // raw MessagePack bytes
)

When verifying integrity, the agent recomputes content_hash for each record and checks that the parent_hash stored in the next record matches. A mismatch at any position indicates that the segment file has been modified after sealing.

Storage sizing

Use the following figures to estimate disk usage:

As a rough rule: for 1 million records with an average payload of 200 bytes, expect approximately 250–280 MB of segment data with encryption enabled.

Configuring storage paths

Set these environment variables before starting the agent to override the default data directories:

SAPIX_STRAND_DIR=/mnt/fast-nvme/strand
SAPIX_GRAPH_DIR=/mnt/fast-nvme/graph
SAPIX_BLOB_DIR=/mnt/object-store/blobs

Backup strategy

Because sealed segments are immutable, an incremental backup only needs to copy newly sealed segments since the last backup run. The active segment and WAL require a coordinated snapshot or a brief agent pause for consistency.

1. Copy all sealed segment files (segment_00000.bin through segment_N-1.bin) — these are safe to copy at any time.
2. Pause writes briefly (or use the snapshot API) to capture the active segment and chain.bin.
3. Back up SAPIX_MASTER_SEED separately in a secrets manager.