> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# High availability & DR

> Running multiple engine replicas, what is safe by design, and recovery objectives

The engine is a single stateless-ish binary in front of one database. That shape makes the HA
story short: scale the database like any Postgres, run N engine replicas behind a load
balancer, and know the three per-process caveats listed below.

## Reference topology

```
            ┌─────────────┐
   SDK ───▶ │ Load        │ ──▶  engine replica 1 ─┐
   OTel ──▶ │ balancer    │ ──▶  engine replica 2 ─┼──▶  Postgres (primary + replica)
   UI ────▶ │ (any L7)    │ ──▶  engine replica N ─┘
            └─────────────┘
```

* **Database**: Postgres via `AGENTX_DB_URL`. SQLite is single-machine by definition - it is
  the right default for a laptop or a single VM, and the wrong choice the moment you want a
  second replica. Tables migrate automatically at boot; replicas racing the same `CREATE TABLE
  IF NOT EXISTS` bootstrap is safe.
* **Replicas**: any number of engines against the same Postgres. No sticky sessions needed:
  the data plane authenticates per request with `x-api-key`, and in `AGENTX_AUTH=enabled` mode
  sessions live in the database, not in process memory.
* **Sizing**: one engine sustains \~900 sequential trace ingests/s (p50 1.1ms) on modest
  hardware; ingest is write-light. Judge features are bounded by your LLM provider, not the
  engine. Start with 2 small replicas for availability rather than throughput.

## What is already multi-replica-safe, by design

* **Background sweeps** (idle-session judging, improvement proposals) take a database lease
  before each tick, so N replicas never judge the same session N times on your API key. This
  is not aspirational: the engine's own test suite runs two lease holders against one database
  and asserts exactly one wins (`sweepLease.test.ts`, plus a Postgres variant).
* **Concurrent ingest** is exercised in CI with 60-way parallel bursts asserting exactly-once
  storage and span dedupe (`concurrency.integration.test.ts`).
* **Migrations** are idempotent (`IF NOT EXISTS` + additive backfills), so rolling restarts
  and replica boots don't race each other destructively.

## The per-process caveats

1. **Rate-limit counters are in-memory per replica.** With N replicas behind a round-robin
   balancer, an abuser gets roughly N times the configured ceiling. Either divide the env
   ceilings (`AGENTX_RATE_LIMIT_CREDENTIAL`, `AGENTX_RATE_LIMIT`) by N, or rate-limit at the
   load balancer and set `AGENTX_RATE_LIMIT=off` inside.
2. **SQLite means one replica.** Everything in this page past the reference topology assumes
   Postgres.
3. **LLM provider keys are shared state in the database** (Platform Settings), so adding a
   replica needs no key redistribution - but a key quota at the provider is shared by all
   replicas together.

## Zero-downtime upgrades

Engines migrate the schema forward at boot and never backward. The safe order:

1. Database-level backup (see [Backup & export](/self-host/backup) - always before upgrades).
2. Roll replicas one at a time: stop one, start the new version, wait for `/health`, repeat.
   Old and new replicas coexisting briefly is fine for additive migrations, which is what the
   engine ships; release notes call out the rare exception.
3. Roll back = restore the pre-upgrade database backup + the old binary. A newer database
   under an older engine is not supported.

## RPO / RTO

| Objective              | With nightly NDJSON + weekly pg\_dump      | With Postgres PITR/streaming replica |
| ---------------------- | ------------------------------------------ | ------------------------------------ |
| RPO (data loss window) | up to 24h                                  | seconds                              |
| RTO (time to restore)  | minutes-hours (restore dump, replay delta) | minutes (promote replica)            |

The pragmatic middle ground for most teams: managed Postgres with point-in-time recovery
turned on, plus the nightly `client.export.dump(since=...)` NDJSON snapshot to object storage
as the vendor-neutral copy that survives even a bad database migration.

## Monitoring the engine itself

* `GET /health` returns `{"status":"ok"}` and is rate-limit-exempt in spirit (data-plane
  ceiling, far above any prober) - point the load balancer's health check here.
* Ingest latency is the canary metric: it is a single indexed insert, so a rising p95 means
  database pressure before anything else notices.
* The [audit trail](/self-host/configuration#the-audit-trail) records config changes and bulk
  exports per replica into the shared database, so the trail stays complete across the fleet.