0022. A watchdog off the control plane pages when the platform can't
Status: accepted, 2026-10-21
Context
Everything that notices trouble runs on the control plane: analytics evaluates rules and notices quiet
services, status runs monitors, notify and its Telegram bot deliver. When that server stops, loses its
network, fills its disk or loses PostgreSQL, or when notify alone stops, nobody is told: the owner, often
away from a laptop with only a phone, hears it from users. Single-server instances have no second machine
at all. Workload nodes are elsewhere, dial out to the control plane (ADR 0008) and reach the internet.
Decision
- notify stays the pager. The watchdog is its fallback and its auditor, not a second alerting system.
- xvii-node runs the watchdog on instance nodes that a superuser gives the
watchdog role (one, or
two in different networks). appsd delivers its settings in the node's desired state: the critical
services to probe at their public /readyz, notify's on-call state endpoint with a read-only token, the
Telegram bot token and the chats to page (the superusers' linked chats by default), and an optional
outside heartbeat URL. The node keeps them and works while the control plane is away.
- Every 30 seconds it looks from outside, like a user. When the control plane is unreachable, or notify
can't deliver (down, or its delivery loop or bot stale), for 3 minutes, it pages Telegram directly
through the Bot API — send-only (
sendMessage, editMessageText; never getUpdates) — with reminders
every 30 minutes and a message when it is back. When a critical service, the AI gateway included, has
been down for 10 minutes while notify is healthy but has no open alert for it, it pages too: the
platform failed to alert. It never pages while its own network is down (Telegram and reference
addresses unreachable).
- Outside heartbeats close the loop. notify requests an optional healthchecks-compatible URL every
minute while it can deliver, and every watchdog requests its own; a third party alerts when they stop:
the control plane and the watchdog failing together, a provider-wide outage, a single-server instance.
- The platform watches the watchdog: its rounds are telemetry, and a seeded rule pages through notify
when a watchdog goes quiet.
- This is the only path that alerts without notify. Its messages carry links, not buttons that would need
the bot to listen; after recovery the node reports the outage through appsd, so it appears in notify's
history.
Consequences
- A bot token reaches nodes. Only instance nodes may have the role (ADR 0008: a node's owner sees what
it is given). The instance bot's token is used send-only by default, so the watchdog works without
anyone creating a bot; a dedicated bot can replace it in notify, and a new token reaches the nodes with
the next desired state.
- A watchdog in the control plane's own network fails with it; a second watchdog elsewhere or an outside
heartbeat is recommended, and notify's Watchdog card says when every watchdog shares that network.
- Two watchdogs may both page for one outage; each says where it saw it.
- Every critical service exposes a readiness endpoint that checks its dependencies and background loops,
shared by the watchdog, status and xvii-node's update gate.
- The architecture's rule that every alert goes through notify gains this one exception.
- Design:
superpowers/specs/2026-10-21-observability-design.md in this repository.