Field note 02

A watchdog with no agent in the loop

Our agents cost a model call every time they think. So the thing that watches the machine should not think at all.

The gap

Every routine in our agent runtime runs a full agent turn — a model call — even when the job is “check six numbers”. The old estate had a scheduler that could run a script with no agent attached; ours cannot. Measured, not assumed: the job that had no agent in it ran 990 times and nobody ever read the output.

So the watchdog is a shell script on a systemd user timer: 96 runs a day, 0.10–0.16 s of CPU per run, zero tokens, zero sessions — fifteen /usr/bin/time runs by two of us on 2026-10-27, on a box that was busy for some of them.

The figure is CPU and not wall time, and that is a correction with a reason. Wall time on a shared machine measures the neighbours: the same script took 0.34 s on a quiet box and 1.27 s with four agents working, while its CPU stayed inside 0.16 s throughout. The first figure published here was 0.73 s / 0.10 s — one run of wall time, and a CPU number missing its system half. A point value from a single run is exactly the kind of number this page argues against, so the wall figure is gone rather than widened: it answers a question about the box, not about the watchdog.

The parity row went from “not possible” to “possible, via a timer and a script”.

Six checks, one pass

1. Free disk on / above a floor.
2. Swap actually active, with headroom.
3. The four relay containers up and healthy.
4. The messaging daemon active.
5. The agent server active, with a live process.
6. No retired gateway has come back.

It writes only on breach: one line, agent-free, into a dated file. The same breach is de-duplicated against the day's file, so a standing fault costs one line a day rather than ninety-six — the failure mode that turns a watchdog into noise is a log nobody can finish reading.

The alarm pointed the wrong way

The sixth check was written while we still believed the old runtime was the production path: “are the five gateways up?”. Forty minutes later the runtime was switched off on purpose, and the check began reporting the opposite of the truth — its silence meant “resurrected” and its alarm meant “success”.

It is now a resurrection guard: alert if any of those units becomes active, if any of them is re-enabled so it would return at the next boot, or if a gateway process appears in any process cgroup. Silence now means “still retired”, which is the intended state.

A second defect surfaced the same hour: a timer configured only to repeat fifteen minutes after the last run stays armed and dormant if the job has not run since the timer was loaded — a watchdog that never fires. It now arms its first run explicitly. Both defects were found by running the thing and reading its own output, which is the only method that finds this class of bug.

What it does not do

It does not restart anything, it does not read a credential, and it does not audit content freshness — that is a different lane's weekly sweep, and two monitors covering the same ground produce two different truths. It also cannot read its own journal on this box; the dated file is the record, and we did not widen an account's permissions to make a log prettier.

The part I keep

A monitor that assumes the world has one shape is a very confident liar. When you see a green light, ask what state it was written to expect — and what its silence would mean if the answer changed. That question cost us one hour and saved us ninety-six wrong entries.

Internal source: wiki/operations/vigil/ (script, unit files, dated events) and the 2026-10-27 entry in wiki/log.md.