SapixDBSapixDB/Docs
Early Access
Phase 5 · Distributed Mode

Distributed Mode

Scale SapixDB horizontally with Cell Division — one command bootstraps a full replica from your existing strand. Choose CP or AP consistency per deployment. Coordinate multi-agent writes safely with Saga transactions.

⬡
Cell Division
One API call replicates every block to a new agent and registers it as a peer. No external orchestrator needed.
⚖
CAP Selectable
Switch between CP (writes block during partition) and AP (writes continue, reconcile on heal) at runtime.
↻
Saga Transactions
Multi-agent writes with automatic compensation. On failure, TOMBSTONE records are sent in reverse order.
⬢
Cross-node Graph
Graph traversal follows edges across cluster nodes transparently. The coordinator fans out sub-queries.

Cell Division

POST /v1/cluster/divide is the single command that creates a replica. SapixDB pushes every existing block to {endpoint}/v1/mesh/ingest, then registers the new agent as a peer. All subsequent writes fan out automatically.

HTTP
POST /v1/cluster/divide
Content-Type: application/json

{ "endpoint": "http://replica-agent:7475" }

# Response
{
  "replica_agent_id": "a7f3c2d1",
  "blocks_pushed": 1842,
  "endpoint": "http://replica-agent:7475"
}
ℹ Bootstrap timeBlock transfer uses the /v1/mesh/ingest internal endpoint and streams all records in genesis → tip order. The replica is immediately readable after the call returns.

Node Roles

Primary
Accepts all writes. Fans out to replicas via the mesh ingest endpoint.
Replica
Full copy. Accepts writes in AP mode; blocks writes in CP mode during partition.
Read Replica
Read-only mirror. Never accepts writes. Good for analytics or backup targets.

CAP Consistency

SapixDB defaults to AP (Available + Partition-tolerant). Switch to CP (Consistent + Partition-tolerant) to block writes cluster-wide when any node becomes unreachable.

AP (default)
  • Writes always succeed on reachable nodes
  • Divergent state reconciled on partition heal
  • Best for high-throughput ingest, IoT, analytics
CP
  • Writes return 503 if any node is unreachable
  • Guarantees all nodes see same strand tip
  • Best for financial ledgers, audit trails, SOX
HTTP
# Switch to CP consistency
POST /v1/cluster/consistency
Content-Type: application/json

{ "consistency": "cp" }

# Response
{ "consistency": "cp", "previous": "ap" }

HTTP API

MethodPathDescription
POST/v1/cluster/divideRegister replica + push all blocks
GET/v1/cluster/nodesList all peer nodes
GET/v1/cluster/statusLive reachability + partition flag
DELETE/v1/cluster/nodes/:agent_idRemove node from mesh
POST/v1/cluster/consistencySwitch CP / AP
POST/v1/sagaCreate and execute saga transaction
GET/v1/sagaList all sagas (most recent first)
GET/v1/saga/:idFetch saga with full step detail

Cluster status response

JSON
{
  "nodes": [
    {
      "agent_id":    "primary-001",
      "endpoint":    "http://primary:7475",
      "role":        "primary",
      "added_at_ms": 1715000000000,
      "last_sync_ms": 1715001234567,
      "reachable":   true
    },
    {
      "agent_id":    "replica-002",
      "endpoint":    "http://replica:7475",
      "role":        "replica",
      "added_at_ms": 1715000050000,
      "last_sync_ms": 1715001230000,
      "reachable":   true
    }
  ],
  "partitioned": false,
  "consistency": "ap"
}

Saga Transactions

A saga is a sequence of writes distributed across multiple agents. If any step fails, SapixDB writes TOMBSTONE compensation records (flags: 2, _saga_compensation: true) to each previously-committed agent in reverse order, preserving strand immutability while logically undoing the effect.

Pending
Transaction created, execution not yet started.
Executing
Steps being applied to target agents in order.
Committed
All steps applied successfully. Transaction is durable.
Compensating
A step failed. Compensating writes being sent in reverse order.
Compensated
All compensation writes written. Side-effects rolled back.
Failed
Compensation also failed. Manual intervention required.
HTTP
POST /v1/saga
Content-Type: application/json

{
  "steps": [
    {
      "target_agent_url": "http://ledger-agent:7475",
      "payload_b64":      "<base64 msgpack>",
      "flags":            0
    },
    {
      "target_agent_url": "http://audit-agent:7475",
      "payload_b64":      "<base64 msgpack>",
      "flags":            0
    }
  ]
}
JSON — committed
{
  "id":                   "a3f8c201",
  "coordinator_agent_id": "primary-001",
  "state":                "committed",
  "created_at_ms":        1715001000000,
  "completed_at_ms":      1715001000042,
  "steps": [
    {
      "step_id":           "0",
      "target_agent_url":  "http://ledger-agent:7475",
      "status":            "applied",
      "record_hash":       "b3d4e5f6a7b8c9d0..."
    },
    {
      "step_id":           "1",
      "target_agent_url":  "http://audit-agent:7475",
      "status":            "applied",
      "record_hash":       "c4e5f6a7b8c9d0e1..."
    }
  ]
}
JSON — compensated (step 1 failed)
{
  "id":    "b2e9f301",
  "state": "compensated",
  "steps": [
    {
      "step_id":           "0",
      "status":            "compensated",
      "compensation_hash": "d5f6a7b8c9d0e1f2..."
    },
    {
      "step_id":           "1",
      "status":            "failed",
      "error":             "connection refused"
    }
  ]
}
⚠ Saga compensation is append-onlyCompensation does not delete records. It writes a new TOMBSTONE record with flags: 2 and _saga_compensation: true in the payload, so the full transaction history — including the rollback — is preserved in the strand.

Cross-node Graph Traversal

Edges can reference agent IDs on any cluster node. When the coordinator receives a GET /v1/graph/traverse request, it resolves which node owns each agent in the traversal path and fans out sub-queries automatically. Results are merged before returning.

HTTP
GET /v1/graph/traverse?from=agent-a&depth=2&direction=outbound

Python SDK

Install the sapixdb-distributed package:

shell
pip install sapixdb-distributed
cluster_setup.py
import asyncio
from sapixdb_distributed import DistributedClient

async def main():
    async with DistributedClient("http://primary:7475") as dist:
        # Replicate to a new node
        result = await dist.divide("http://replica:7475")
        print(f"Replica {result.replica_agent_id}: {result.blocks_pushed} blocks synced")

        # Check health
        status = await dist.cluster_status()
        for node in status.nodes:
            mark = "✓" if node.reachable else "✗"
            print(f"  {mark} {node.agent_id} ({node.role})")

        # Switch to CP consistency for a SOX-regulated window
        await dist.use_cp()
        print("Switched to CP consistency")

asyncio.run(main())
saga_example.py
import asyncio
from sapixdb_distributed import DistributedClient, SAGA_COMMITTED

async def transfer(amount: float, from_url: str, to_url: str):
    async with DistributedClient(from_url) as dist:
        saga = await dist.execute_saga([
            {
                "target_agent_url": from_url,
                "payload": {"action": "debit", "amount": amount, "currency": "USD"},
            },
            {
                "target_agent_url": to_url,
                "payload": {"action": "credit", "amount": amount, "currency": "USD"},
            },
        ])

        if saga.state == SAGA_COMMITTED:
            print(f"Transfer committed: saga {saga.id}")
        else:
            print(f"Transfer rolled back: {saga.state}")
            for step in saga.steps:
                if step.error:
                    print(f"  Step {step.step_id}: {step.error}")

asyncio.run(transfer(100.0, "http://ledger-a:7475", "http://ledger-b:7475"))
via SapixClient (no extra package)
from sapixdb_agent import SapixClient
import base64, msgpack

async def main():
    async with SapixClient("http://primary:7475") as client:
        # Cluster
        nodes = await client.cluster_nodes()
        status = await client.cluster_status()

        # Saga (encode payloads yourself)
        def encode(obj):
            return base64.b64encode(msgpack.packb(obj, use_bin_type=True)).decode()

        saga = await client.saga_execute([
            {"target_agent_url": "http://agent-a:7475", "payload_b64": encode({"x": 1}), "flags": 0},
            {"target_agent_url": "http://agent-b:7475", "payload_b64": encode({"x": 2}), "flags": 0},
        ])
        print(saga["state"])

SDK API Reference

MethodReturnsDescription
divide(endpoint)DivideResponseRegister replica and push all blocks
list_nodes()ClusterNodesResponseReturn all registered cluster nodes
cluster_status()ClusterStatusResponseLive reachability check + partition flag
remove_node(agent_id)NoneRemove node from mesh
set_consistency(mode)ConsistencyResponseSwitch 'cp' or 'ap'
use_cp()ConsistencyResponseShorthand for CP consistency
use_ap()ConsistencyResponseShorthand for AP consistency
is_healthy()boolTrue if no unreachable nodes and not partitioned
execute_saga(steps)SagaTransactionEncode payloads and run distributed saga
list_sagas()SagaListResponseAll sagas, most recent first
get_saga(saga_id)SagaTransactionSingle saga with full step detail

Limitations

Kafka and PostgreSQL CDC inbound workers are config-only in Phase 4/5. Active consumption ships in a later release.
Partition healing (divergence reconciliation in AP mode) is not yet implemented. Reachable nodes always stay in sync; network splits require manual re-division.
Cross-node graph traversal fan-out is best-effort. If a remote node is unreachable, that branch of the traversal is omitted from results.
Saga compensation is append-only and preserves full history. It does not physically delete or modify previously written records.
Scale your SapixDB deployment
Cell Division, saga transactions, and selectable CAP consistency — all available in early access.
Request Early Access