SapixDBSapixDB/Docs
Home
Phase 5 · Distributed Mode

Distributed Mode

◆ EnterpriseRequires the enterprise build. Community builds return 404 on these endpoints.

Scale SapixDB horizontally with Cell Division — one command bootstraps a full replica from your existing strand. Choose CP or AP consistency per deployment. Coordinate multi-agent writes safely with Saga transactions.

⬡
Cell Division
One API call replicates every block to a new agent and registers it as a peer. No external orchestrator needed.
⚖
CAP Selectable
Switch between CP (writes block during partition) and AP (writes continue, reconcile on heal) at runtime.
↻
Saga Transactions
Multi-agent writes with automatic compensation. On failure, TOMBSTONE records are sent in reverse order.
⬢
Cross-node Graph
Graph traversal follows edges across cluster nodes transparently. The coordinator fans out sub-queries.

Cell Division

POST /v1/cluster/divide is the single command that creates a replica. SapixDB pushes every existing block to {endpoint}/v1/mesh/ingest, then registers the new agent as a peer. All subsequent writes fan out automatically.

HTTP
POST /v1/cluster/divide
Content-Type: application/json

{ "endpoint": "http://replica-agent:7475" }

# Response
{
  "replica_agent_id": "a7f3c2d1",
  "blocks_pushed": 1842,
  "endpoint": "http://replica-agent:7475"
}
ℹ Bootstrap timeBlock transfer uses the /v1/mesh/ingest internal endpoint and streams all records in genesis → tip order. The replica is immediately readable after the call returns.
⚠ Cluster endpoints require admin:clusterPOST /v1/cluster/divide, POST /v1/cluster/consistency, and DELETE /v1/cluster/nodes/:agent_id require the root key or a scoped key holding admin:cluster — a plain write:* key is not enough. See Scopes.
ℹ Authenticating node-to-node calls — SAPIX_MESH_TOKENIn a deployment where every node has API-key auth configured, the internal calls one node makes to another — pushing replication blocks, pull-syncing the strand, cross-node graph traversal — need a credential too, or the peer rejects them. Set the same SAPIX_MESH_TOKEN on every node in the cluster; SapixDB sends it as Authorization: Bearer $SAPIX_MESH_TOKEN on every peer-to-peer call and accepts a matching token as root-equivalent (registering a peer already implies that level of mutual trust — a peer can push arbitrary blocks into your strand). Leave it unset in an all-open-mode cluster; node-to-node calls carry no credential and nothing changes.
⚠ Protect SAPIX_MESH_TOKEN like a root keyIt's root-equivalent on every node that accepts it — store it in a secrets manager, not a plain .env file, and never pass it on a shell command line where it can land in process listings or shell history. A leaked mesh token grants full root access from any node in the cluster, not just peer-to-peer replication.

Node Roles

Primary
Accepts all writes. Fans out to replicas via the mesh ingest endpoint.
Replica
Full copy. Accepts writes in AP mode; blocks writes in CP mode during partition.
Read Replica
Read-only mirror. Never accepts writes. Good for analytics or backup targets.

CAP Consistency

SapixDB defaults to AP (Available + Partition-tolerant). Switch to CP (Consistent + Partition-tolerant) to block writes cluster-wide when any node becomes unreachable.

AP (default)
  • Writes always succeed on reachable nodes
  • Divergent state reconciled on partition heal
  • Best for high-throughput ingest, IoT, analytics
CP
  • Writes return 503 if any node is unreachable
  • Guarantees all nodes see same strand tip
  • Best for financial ledgers, audit trails, SOX
HTTP
# Switch to CP consistency
POST /v1/cluster/consistency
Content-Type: application/json

{ "consistency": "cp" }

# Response
{ "consistency": "cp", "previous": "ap" }

HTTP API

MethodPathDescription
POST/v1/cluster/divideRegister replica + push all blocks
GET/v1/cluster/nodesList all peer nodes
GET/v1/cluster/statusLive reachability + partition flag
DELETE/v1/cluster/nodes/:agent_idRemove node from mesh
POST/v1/cluster/consistencySwitch CP / AP
POST/v1/sagaCreate and execute saga transaction
GET/v1/sagaList all sagas (most recent first)
GET/v1/saga/:idFetch saga with full step detail

Cluster status response

JSON
{
  "nodes": [
    {
      "agent_id":    "primary-001",
      "endpoint":    "http://primary:7475",
      "role":        "primary",
      "added_at_ms": 1715000000000,
      "last_sync_ms": 1715001234567,
      "reachable":   true
    },
    {
      "agent_id":    "replica-002",
      "endpoint":    "http://replica:7475",
      "role":        "replica",
      "added_at_ms": 1715000050000,
      "last_sync_ms": 1715001230000,
      "reachable":   true
    }
  ],
  "partitioned": false,
  "consistency": "ap"
}

Saga Transactions

A saga is a sequence of writes distributed across multiple agents. If any step fails, SapixDB writes TOMBSTONE compensation records (flags: 2, _saga_compensation: true) to each previously-committed agent in reverse order, preserving strand immutability while logically undoing the effect.

Pending
Transaction created, execution not yet started.
Executing
Steps being applied to target agents in order.
Committed
All steps applied successfully. Transaction is durable.
Compensating
A step failed. Compensating writes being sent in reverse order.
Compensated
All compensation writes written. Side-effects rolled back.
Failed
Compensation also failed. Manual intervention required.
HTTP
POST /v1/saga
Content-Type: application/json

{
  "steps": [
    {
      "target_agent_url": "http://ledger-agent:7475",
      "payload_b64":      "<base64 msgpack>",
      "flags":            0
    },
    {
      "target_agent_url": "http://audit-agent:7475",
      "payload_b64":      "<base64 msgpack>",
      "flags":            0
    }
  ]
}
JSON — committed
{
  "id":                   "a3f8c201",
  "coordinator_agent_id": "primary-001",
  "state":                "committed",
  "created_at_ms":        1715001000000,
  "completed_at_ms":      1715001000042,
  "steps": [
    {
      "step_id":           "0",
      "target_agent_url":  "http://ledger-agent:7475",
      "status":            "applied",
      "record_hash":       "b3d4e5f6a7b8c9d0..."
    },
    {
      "step_id":           "1",
      "target_agent_url":  "http://audit-agent:7475",
      "status":            "applied",
      "record_hash":       "c4e5f6a7b8c9d0e1..."
    }
  ]
}
JSON — compensated (step 1 failed)
{
  "id":    "b2e9f301",
  "state": "compensated",
  "steps": [
    {
      "step_id":           "0",
      "status":            "compensated",
      "compensation_hash": "d5f6a7b8c9d0e1f2..."
    },
    {
      "step_id":           "1",
      "status":            "failed",
      "error":             "connection refused"
    }
  ]
}
⚠ Saga compensation is append-onlyCompensation does not delete records. It writes a new TOMBSTONE record with flags: 2 and _saga_compensation: true in the payload, so the full transaction history — including the rollback — is preserved in the strand.

Cross-node Graph Traversal

Edges can reference agent IDs on any cluster node. When the coordinator receives a GET /v1/graph/traverse request, it resolves which node owns each agent in the traversal path and fans out sub-queries automatically. Results are merged before returning.

HTTP
GET /v1/graph/traverse?agent_id=agent-a&depth=2
JSON — one peer unreachable
{
  "start_agent": "agent-a",
  "agents_visited": ["agent-a", "agent-b"],
  "edges": [ /* ... */ ],
  "depth_reached": 1,
  "partial": true,
  "unreachable_agents": ["agent-c"]
}
ℹ A failed peer is reported, not silently omittedIf a remote node's edges can't be fetched — network failure, or a 401/403 from a peer that requires SAPIX_MESH_TOKEN and didn't get one that matches — partial is true and the affected agent is named in unreachable_agents. The traversal is genuinely complete for every agent not listed there.

Python SDK

Install the sapixdb-distributed package:

shell
pip install sapixdb-distributed
cluster_setup.py
import asyncio
from sapixdb_distributed import DistributedClient

async def main():
    async with DistributedClient("http://primary:7475") as dist:
        # Replicate to a new node
        result = await dist.divide("http://replica:7475")
        print(f"Replica {result.replica_agent_id}: {result.blocks_pushed} blocks synced")

        # Check health
        status = await dist.cluster_status()
        for node in status.nodes:
            mark = "✓" if node.reachable else "✗"
            print(f"  {mark} {node.agent_id} ({node.role})")

        # Switch to CP consistency for a SOX-regulated window
        await dist.use_cp()
        print("Switched to CP consistency")

asyncio.run(main())
saga_example.py
import asyncio
from sapixdb_distributed import DistributedClient, SAGA_COMMITTED

async def transfer(amount: float, from_url: str, to_url: str):
    async with DistributedClient(from_url) as dist:
        saga = await dist.execute_saga([
            {
                "target_agent_url": from_url,
                "payload": {"action": "debit", "amount": amount, "currency": "USD"},
            },
            {
                "target_agent_url": to_url,
                "payload": {"action": "credit", "amount": amount, "currency": "USD"},
            },
        ])

        if saga.state == SAGA_COMMITTED:
            print(f"Transfer committed: saga {saga.id}")
        else:
            print(f"Transfer rolled back: {saga.state}")
            for step in saga.steps:
                if step.error:
                    print(f"  Step {step.step_id}: {step.error}")

asyncio.run(transfer(100.0, "http://ledger-a:7475", "http://ledger-b:7475"))
via SapixClient (no extra package)
from sapixdb_agent import SapixClient
import base64, msgpack

async def main():
    async with SapixClient("http://primary:7475") as client:
        # Cluster
        nodes = await client.cluster_nodes()
        status = await client.cluster_status()

        # Saga (encode payloads yourself)
        def encode(obj):
            return base64.b64encode(msgpack.packb(obj, use_bin_type=True)).decode()

        saga = await client.saga_execute([
            {"target_agent_url": "http://agent-a:7475", "payload_b64": encode({"x": 1}), "flags": 0},
            {"target_agent_url": "http://agent-b:7475", "payload_b64": encode({"x": 2}), "flags": 0},
        ])
        print(saga["state"])

SDK API Reference

MethodReturnsDescription
divide(endpoint)DivideResponseRegister replica and push all blocks
list_nodes()ClusterNodesResponseReturn all registered cluster nodes
cluster_status()ClusterStatusResponseLive reachability check + partition flag
remove_node(agent_id)NoneRemove node from mesh
set_consistency(mode)ConsistencyResponseSwitch 'cp' or 'ap'
use_cp()ConsistencyResponseShorthand for CP consistency
use_ap()ConsistencyResponseShorthand for AP consistency
is_healthy()boolTrue if no unreachable nodes and not partitioned
execute_saga(steps)SagaTransactionEncode payloads and run distributed saga
list_sagas()SagaListResponseAll sagas, most recent first
get_saga(saga_id)SagaTransactionSingle saga with full step detail

Limitations

Kafka and PostgreSQL CDC inbound workers are config-only in Phase 4/5. Active consumption ships in a later release.
Partition healing (divergence reconciliation in AP mode) is not yet implemented. Reachable nodes always stay in sync; network splits require manual re-division.
Cross-node graph traversal fan-out is best-effort. If a remote node is unreachable, that branch is omitted from results — but the response reports it via partial: true and unreachable_agents, it's not a silent gap.
Saga compensation is append-only and preserves full history. It does not physically delete or modify previously written records.
Scale your SapixDB deployment
Cell Division, saga transactions, and selectable CAP consistency — available now.
Start Free Trial →