Skip to main content
The engine is one near-stateless binary in front of your database(s). That shape makes the availability story short: make the database highly available like any Postgres, keep the engine restartable in seconds, and know the one hard topology rule below.

​
The topology rule: one engine instance

The engine runs as a single instance in every tier - it is the one telemetry writer. The ingest queue micro-batches and dedupes per process, and the ClickHouse adapter enforces span idempotency adapter-side on the same assumption; the Helm chart pins replicas: 1 with a Recreate strategy for exactly this reason. Multi-writer ingest is a named future ADR, not a --set replicas=3 away. What you scale instead is the storage tier: A single engine process is not the bottleneck it sounds like: the measured baseline (engine/scripts/bench.mjs, 32 concurrent senders, production-shaped payloads) sustains ~2,300 stored spans/s with an ingest ack p95 of 18 ms and dashboard query p95 of 8 ms - at that concurrency the HTTP layer, not storage, is the bound. Judge features are bounded by your LLM provider, not the engine. CI re-measures this nightly with regression floors, and a weekly chaos suite drills the failure modes in the repo’s engine/docs/runbook.md.

​
What makes restarts and redeploys safe by design

Availability for a single-instance engine means fast, safe restarts, and the engine is built for them:
  • Graceful shutdown: SIGTERM/SIGINT stop accepting connections, drain the ingest queue, flush, and close the database - including signals that land mid-boot.
  • Idempotent migrations run at boot (IF NOT EXISTS + additive backfills), so a restart or a brief old/new overlap during a redeploy never races destructively.
  • Lease-protected background sweeps (idle-session judging, improvement proposals) take a database lease before each tick, so an overlapping old and new process during a handoff never judges the same session twice on your API key (sweepLease.test.ts, plus a Postgres variant, prove it with two lease holders against one database).
  • Client-side buffering: the SDK queues and retries with backoff on 429/503, and span ids make redelivery idempotent - a restart window loses nothing that a client was still holding.

​
Upgrades

Engines migrate the schema forward at boot and never backward. The safe order:
  1. Database-level backup (see Backup & export - always before upgrades).
  2. Stop the old engine, start the new one, wait for GET /health. With Recreate semantics this is a seconds-long gap; SDKs ride it out on their retry queue.
  3. Roll back = restore the pre-upgrade database backup + the old binary. A newer database under an older engine is not supported.

​
RPO / RTO

The pragmatic middle ground for most teams: managed Postgres with point-in-time recovery turned on, plus the nightly client.export.dump(since=...) NDJSON snapshot to object storage as the vendor-neutral copy that survives even a bad database migration. In the ClickHouse tier, add span backups per Backup & export - spans are append-only, so incrementals are cheap.

​
Monitoring the engine itself

  • GET /health returns {"status":"ok"} - point your orchestrator’s liveness/health check here (it sits under the data-plane rate ceiling, far above any prober).
  • GET /metrics serves Prometheus metrics: queue depth, stored/deduped/rejected/dropped counters, RSS, uptime. agentx_ingest_spans_dropped_total above zero is an incident; sustained 429s (agentx_ingest_spans_rejected_total climbing) mean the storage backend cannot keep up - raising AGENTX_INGEST_QUEUE_MAX buys burst absorption, not throughput.
  • Ingest latency is the canary metric: it is a micro-batched insert, so a rising ack p95 means database pressure before anything else notices.
  • If ClickHouse goes down (enterprise tier), ingest answers 503 + Retry-After and clients redeliver; the control plane and dashboard stay up, and ingestion resumes on recovery with no operator action.
  • The audit trail records config changes and bulk exports into the database, so the trail survives engine restarts and redeploys.

​
Rate limits behind a load balancer

Rate-limit counters are in-memory and per-IP. Behind a proxy or load balancer, either terminate rate limiting there and set AGENTX_RATE_LIMIT=off inside, or make sure the engine sees real client IPs - otherwise every client shares the proxy’s counter.