0024. The disk guard reclaims rebuildable data on every server
Status: accepted, 2026-10-21. Refines the rule of ADR 0015 that the agent leaves anything without xvii
labels alone.
Context
Build caches and images fill a disk in minutes, and when a disk fills, PostgreSQL on it goes into
recovery with every database it holds. A static "disk over 90%" rule fires late and frees nothing; the
clean-up afterwards was done by hand or by scripts on single servers. xvii-node prunes its own built
images, but not build cache, unused images, logs, or the control plane's installed versions. ADR 0015
promises that the agent leaves alone anything without xvii labels, and some servers are shared with
software that isn't xvii's (a CI runner, an adopted compose project); Docker's build cache carries no
owner at all.
Decision
- xvii-node runs a disk guard on every server, the control plane included. It measures free space and
inodes every minute, forecasts the time to full from the last hour and the last six hours, and keeps a
reserve free: the larger of 15% and 10 GiB by default, at most 30% of the disk.
- Under the reserve, or filling fast, it frees space in fixed tiers, only of data that is rebuilt or
unreferenced: its own first (builds beyond the rollback depth, crashed build folders, installed
versions and releases beyond what rollback keeps), then Docker's build cache older than a day and
dangling images, then all build cache and images no workload or rollback release uses, then the journal
down to a cap. A ballast file goes first in an emergency and comes back when there is room.
- Ownership decides the defaults. Where Docker is the agent's alone (the join script installed it),
the Docker and journal tiers are on; on a shared server they stay off until the node's manager turns
them on. Containers, networks, volumes, images a desired workload or its rollback uses, databases and
backups are never touched, on any server.
- Below a hard floor (5% by default) builds are refused; deploys are not. Every reclaim is recorded with
what it freed, the forecast is reported as telemetry, and analytics alerts on it per server.
Consequences
- ADR 0015's rule holds for containers, networks and volumes; Docker caches without labels and the
journal become reclaimable where the node's manager allows it.
- Losing build cache makes the next builds slower; tiers stop as soon as the reserve is back.
- The control plane keeps the current and three newest installed versions of each service, so binary
rollbacks reach that far; home's release store keeps five.
- Design:
superpowers/specs/2026-10-21-observability-design.md in this repository.