0023. Alerts are keyed problems; notify pages them until someone acknowledges

Status: accepted, 2026-10-21

Context

Alerting grew per service. analytics evaluates each rule over all its series at once, so a CPU rule averages every server and can never fire for one, and messages don't say which server or service. status sends an event per monitor, so one resolver failure on the probe is dozens of "down". Services emit their own problem events, each in its own shape. notify sends every event once, with an Acknowledge button that stops nothing because nothing repeats; a recovery below a subscription's minimum severity never arrives. Static thresholds are noisy on busy services and blind on quiet ones. The result is fatigue: the real alert at night is easy to miss.

Decision

Sources detect, notify pages. A problem is detected where its data lives: analytics (rules on metrics and logs, SLO burn rates), status (monitors and journeys), a service (a deploy failed, a node went offline). The source keeps the problem's state and sends one event when it starts and one when it ends, carrying an alert key (stable for that problem within the service), a state (firing, resolved), a group and labels (node, service, app). Rules group by labels, so a problem belongs to a server or a service, never to an average.
notify turns firing problems into alerts: problems of one group (node:n1, service:ai) are one alert and page once; a problem that flaps within minutes reopens its alert instead of paging again; storms are capped per channel.
Policies page until acknowledged. Per team, and one for the instance: critical pages, warning notifies once, info goes to the feed. A page repeats and escalates along the policy's steps (more channels, more people) until someone acknowledges, snoozes or resolves it, or its sources resolve it. Acknowledging works from the panel, Telegram, push, the CLI and MCP. The resolution reaches every channel that heard the problem, whatever its subscription. Silences (matchers with an end) hold pages back.
SLOs page, thresholds warn. Where an SLO exists, fast and medium error-budget burn over two windows pages; static thresholds stay as floors and warnings.

Consequences

kit/events.Event gains alert; every service maps its problem and recovery events onto keys and groups, and lists them in its spec. notify must accept the field before any service sends it (unknown fields are refused), so it is upgraded first.
notify keeps state about problems (alerts, policies, on-call people, silences) and is the one place people acknowledge; home's Today, the PWA and the native apps read it.
Detection quality is each source's job: status tells its own blind spots (a probe that can't resolve names) from targets being down; analytics validates rules and reports a rule it can't evaluate.
status keeps "incidents" for public status pages; notify's object is an "alert", and the two link.
Design: superpowers/specs/2026-10-21-observability-design.md in this repository.