Operations

Running it: what to watch, what to do when something is wrong, and the one irreversible mistake.

Never run docker compose down -v

It drops the volume, and the volume is where the paired WhatsApp devices live. Each number comes back only when a human re-scans a QR with the physical phone. Put deploy/backup.sh on a cron so the mistake is recoverable.

Health checks#

Endpoint	Question	On failure
GET /v1/live	Is the process up?	Restart it.
GET /v1/ready	Can it serve? 503 when Postgres is unreachable.	Take it out of rotation. Do NOT restart — a database briefly unreachable is not fixed by killing every process that noticed.
GET /v1/admin/alerts	Is anything actually wrong? 503 while a critical rule fires.	Read the alerts array; each carries the action to take.

Point the liveness probe and the load balancer at different endpoints

Wiring both to the same one is how a database blip becomes a restart storm.

Being told, rather than having to ask#

/metrics is a pull endpoint, so every number on it is worth exactly as much as somebody’s habit of looking — and the failures worth catching are the quiet ones.

shell

curl -s https://switchboard.atrix.dev/v1/admin/alerts -H "apikey: $GLOBAL_API_KEY" | jq

Six rules, all reported whether firing or not — “all clear” and “the evaluator broke” must not look the same. Set ALERT_WEBHOOK_URL to be told instead of asking; Slack, Discord and Mattermost work unmodified.

Transitions only

Once when a condition starts, once when it clears, and nothing in between however long it lasts. A version that re-sent every minute would produce a webhook a minute for three days, and the response to that is always to mute the channel — after which there is a monitor everyone believes in and nobody reads.

Did we lose an inbound message?#

No — unless it is in the dead-letter queue, which is exactly what that queue is for.

shell

curl -s "https://switchboard.atrix.dev/v1/admin/deliveries?state=dead" \
  -H "apikey: $GLOBAL_API_KEY" | jq '.deliveries[] | {id, event, lastError, attempts}'

curl -X POST https://switchboard.atrix.dev/v1/admin/deliveries/<id>/replay \
  -H "apikey: $GLOBAL_API_KEY"

A delivery is retried at 10s, 30s, 2m, 5m, 15m, 1h and 6h — well past any deploy. Only after all of it does the event become a dead letter.

A dead letter is replayable for 30 days, not forever

The row is kept indefinitely — it is the record of what was lost — but the payload is a customer’s message, so it expires. After that the row remains and replay answers 410.

Running more than one replica#

A paired number lives in exactly one process — its websocket and protocol client are held in memory there. Sharing that between replicas is what sharding does.

env

SHARDING_ENABLED=true
REPLICA_URL=http://switchboard-1:4000    # reachable from the OTHER replicas

Do not run two replicas without it

The second is not merely useless, it is destructive: a send landing on the replica that does not hold the socket starts its own WhatsApp client against the same device credentials, and the two evict each other until the number is unusable — while both log successful reconnections.

With it on, a replica claims instances with a lease. One rule carries the design: a replica only ever opens a socket for an instance it owns. Requests landing elsewhere are forwarded to the owner, so consumers never learn any of this exists.