Skip to main content
The engine is a single stateless-ish binary in front of one database. That shape makes the HA story short: scale the database like any Postgres, run N engine replicas behind a load balancer, and know the three per-process caveats listed below.

​
Reference topology

  • Database: Postgres via AGENTX_DB_URL. SQLite is single-machine by definition - it is the right default for a laptop or a single VM, and the wrong choice the moment you want a second replica. Tables migrate automatically at boot; replicas racing the same CREATE TABLE IF NOT EXISTS bootstrap is safe.
  • Replicas: any number of engines against the same Postgres. No sticky sessions needed: the data plane authenticates per request with x-api-key, and in AGENTX_AUTH=enabled mode sessions live in the database, not in process memory.
  • Sizing: one engine sustains ~900 sequential trace ingests/s (p50 1.1ms) on modest hardware; ingest is write-light. Judge features are bounded by your LLM provider, not the engine. Start with 2 small replicas for availability rather than throughput.

​
What is already multi-replica-safe, by design

  • Background sweeps (idle-session judging, improvement proposals) take a database lease before each tick, so N replicas never judge the same session N times on your API key. This is not aspirational: the engine’s own test suite runs two lease holders against one database and asserts exactly one wins (sweepLease.test.ts, plus a Postgres variant).
  • Concurrent ingest is exercised in CI with 60-way parallel bursts asserting exactly-once storage and span dedupe (concurrency.integration.test.ts).
  • Migrations are idempotent (IF NOT EXISTS + additive backfills), so rolling restarts and replica boots don’t race each other destructively.

​
The per-process caveats

  1. Rate-limit counters are in-memory per replica. With N replicas behind a round-robin balancer, an abuser gets roughly N times the configured ceiling. Either divide the env ceilings (AGENTX_RATE_LIMIT_CREDENTIAL, AGENTX_RATE_LIMIT) by N, or rate-limit at the load balancer and set AGENTX_RATE_LIMIT=off inside.
  2. SQLite means one replica. Everything in this page past the reference topology assumes Postgres.
  3. LLM provider keys are shared state in the database (Platform Settings), so adding a replica needs no key redistribution - but a key quota at the provider is shared by all replicas together.

​
Zero-downtime upgrades

Engines migrate the schema forward at boot and never backward. The safe order:
  1. Database-level backup (see Backup & export - always before upgrades).
  2. Roll replicas one at a time: stop one, start the new version, wait for /health, repeat. Old and new replicas coexisting briefly is fine for additive migrations, which is what the engine ships; release notes call out the rare exception.
  3. Roll back = restore the pre-upgrade database backup + the old binary. A newer database under an older engine is not supported.

​
RPO / RTO

The pragmatic middle ground for most teams: managed Postgres with point-in-time recovery turned on, plus the nightly client.export.dump(since=...) NDJSON snapshot to object storage as the vendor-neutral copy that survives even a bad database migration.

​
Monitoring the engine itself

  • GET /health returns {"status":"ok"} and is rate-limit-exempt in spirit (data-plane ceiling, far above any prober) - point the load balancer’s health check here.
  • Ingest latency is the canary metric: it is a single indexed insert, so a rising p95 means database pressure before anything else notices.
  • The audit trail records config changes and bulk exports per replica into the shared database, so the trail stays complete across the fleet.