Of 25 failures in four weeks, my monitoring caught and acted on four. On checking the thing, not the claim, and on who checks the checks.
On 2 September I had Claude run a full sweep of my system: five read-only audits, including a look back at ninety days of failures. My notes record that the storage audit credited my document archive with a second copy: a weekly job copied it to the home server.
That evening the job itself was checked. It had never once succeeded. Every Sunday since mid-July it had started on time, tried to create a folder it had no permission for, and stopped before opening its log. No error, no alert.
Four of twenty-five
A retrospective on 5 August had found 25 evidenced failures in the four weeks from 7 July. A machine caught six: four it acted on before anyone noticed, two that nobody acted on. A person noticed seven; twelve were found by accident. Two jobs meanwhile sent about 56 false failure alerts in eight days.
That night a headless Claude run (nobody at the keyboard) on Yavin* built a regression suite: 136 weekly checks for what used to work and no longer does. A development task is now done only when a run is green or its red results are explained.
Active, said systemd
On 26 July the Telegram gateway, my alert channel, was dead for about twelve hours; on 28 July the notes sync hung for about fifteen. systemd (the Linux service manager) called both active throughout: it knows whether a process is alive, not whether the work gets done.
On 29 July Claude built a watchdog that looks for evidence of work, such as the sync’s “fully synced” log line. For the gateway it checked only the process, which from 30 July lived on for two days, idle.
Claims written down as done
Some green ticks are written by hand, often by Claude. On 13 May the morning stand-up reported 129 overdue tasks; the real number was zero.
In September I moved my automations’ secrets to a store that machines read unattended. On 17 September its fallback tests (the path taken when the store is unreachable) passed seven of seven in 0.00 seconds each: they had never touched the network, and Claude fixed them that session. That afternoon the task note called the store “flipped live … verified for real”. A correction on 25 September said it had been live for no consumer on two machines: the health check had read the merged configuration, that is, the claim. The fallback meant nothing broke. On 9 October the switch was verified by turning the store off and on and watching what the machines loaded.
Ninety days, and a monitor for absences
The sweep found that my three most important hard rules had held: zero violations in 635 archived sessions. The damage was in things that stopped silently, such as a watcher that crashed every two hours for nine days in July and could not report its own death.
On 3 September Claude built an absence monitor: a list of expected jobs and their rhythms, checked weekly for a trace of each. It has been wrong both ways: on 2 October it passed a failing job, because it checked only that the timer had fired; on 9 October it flagged three jobs that had run, because the weekly reboot had wiped the evidence. Both are fixed.
The pattern
Check the thing, not the claim; nothing watches the watchers; and a scheduled job is done when its timer has fired, not when it worked once by hand. A claim can come from anywhere: a service state, a passing test, an audit, a task note an hour old. Every check is a claim too, and must be able to go red for the right reason. In July, two backup moves written up as running were one-time copies.
Claude wrote many of these claims, and also built the checks, caught the 0.00 seconds and wrote the corrections. I now ask for “verified how” next to “done”.
It is not finished. The chain ends with me reading a weekly regression summary on Telegram; the last before I wrote this was red. Nothing alerts when the home server itself stops answering: every watchdog runs on it.
Part 6 of twelve in the series on how I built my AI OS. Start with the preface.
Next: a rule written in capital letters that broke again anyway, and where rules really live.
* Host names are changed (here: Yavin, my home server). The real ones come from a different saga.
Appendix — for builders
- Count how your failures were found. Caught by a machine and acted on, caught and ignored, noticed by a person, found by accident: mine were 4, 2, 7 and 12 of 25.
- Check evidence of work, not a live process. A heartbeat line in the log, a changed file, an arrived message.
- Register the check in the same change as the job. List the expected jobs and their rhythm; a job that leaves no trace in its window is a finding. My newest watcher died two days after it shipped and got its own check three weeks later.
- Distrust a test that is too fast. Make each new check go red once, for the right reason, before trusting its green.
- Write “verified how” next to “done”. For scheduled work: the timer exists, has fired twice, and a monitor saw the second run.
- Give the chain a human end. One line you expect each week; if it does not come, the checker is the failure.
- What I considered instead: more alerts on the monitor I had (it already sent mostly false ones), systemd’s restart rules (they fire only when a process exits; these did not), and an occasional big audit (the September sweep found the archive copy seven weeks late). I chose a weekly regression suite and an absence monitor: both check evidence on a schedule, and a weekly line I read keeps them honest.