Autonomous agents don't crash. They go quiet and keep reporting success.
This is a field catalog of that failure class, taken from a small company whose operations actually run on scheduled AI agents: 439 cron jobs, of which 159 exist only to catch the other 280 lying, and 336 written post-incident notes accumulated since June 2026.
Everything here is a real incident with a real date. Nothing is hypothetical.
The runnable checks in guards/ are the generic version of what we
now run in production.
A guard you have never watched fail is not a guard. It is a comment.
On 2026-08-18 we audited five of our own safety checks at once. All five were fake:
| The check | What it claimed | What it actually did |
|---|---|---|
| pre-deploy host probe | measures which host serves live | defaulted to one of two answers; had never probed one of them. Right 5 times out of 8, by luck |
| unapproved-send verifier | proves zero unapproved sends | built an empty list, never filled it, so pass was permanently True |
| room classifier | labels each conversation | wrote unknown as a valid record when the model errored, so "never looked at" became "looked, can't tell" — permanently |
| disk guard | halt above 88% | measured a denominator that excluded the root reserve; the real trigger was 93% |
| decision→reply loop | collect human verdicts | the collection code did not exist. 32 of 32 verdicts were given and 0 were captured |
They share one shape: there is no path on which they fail. Every one of them falls back to something that looks like success. So when they broke, nothing made a sound.
The fix is not more checks. It is: break your check on purpose, watch the alarm
fire, and only then call it wired. See guards/guard-selftest.py.
2026-08-22. A document verifier showed "12 checks passed" and looked healthy. Two of its checks were printing neither pass nor fail. One had stopped matching because a table cell gained a CSS class; the other because we ourselves had edited the wording it was matching against. Our own edit switched off our own guard.
A failing check makes noise. A check that isn't running makes none, and a pass counter cannot tell the two apart.
Apply: make "expected to find this, found nothing" a failure, not a pass. Alarm on the count of executed checks changing at all — 12→14 meant a check came back to life, and we needed to know why.
Variant, 2026-08-25: the checker built to catch this trap silently passed
itself. GNU grep -E invoked from inside a shell script failed to match a
[가-힣] character range under that locale, while the same command typed by hand
worked. We planted a deliberately bad file and got a green tick.
→ Never write the tool that judges a language in that same language. Our
shell-script checker is now written in Node.
2026-08-18. Every session start printed 868 classifier failures · 100% silent.
The classifier was fine — running on a 2-minute schedule, exit 0, heartbeat
fresh. The crash was one old event, already fixed.
868 came from one line: len(re.findall("classifier failed", log_text)) over
the entire append-only log. It could never go down.
Apply: count alerts over a recent window, and rotate the log the counter reads. A cumulative counter guarantees a false alarm eventually, and alarm fatigue is what buries the real one.
Second face of the same trap, 2026-08-24: the window was right, but it
counted attempts rather than unresolved ones. A send-ledger watchdog cried
"11 failures in 24h" all night. Actual customers who never received anything:
zero. The 11 were 3 of our own test rows, 3 ghost rows the watchdog itself
had cleaned up, and 5 first attempts that later succeeded on retry.
→ Count "never got there", not "tried and missed."
See guards/recent-failures.py.
2026-08-25. A stale-lead watcher logged "0 stalled" every day. There were five leads stalled for 7, 8 and 10 days. The query was correct. The culprit was a 3-day re-alert cooldown: it alerted once, wrote itself a note, and went silent while nobody acted.
A guard that alerts once and then hushes cannot distinguish resolved from nobody moved. You get the exact inversion you want least: the most-rotten item is the quietest.
Apply: make the cooldown shrink as the delay grows (ours: past 7 days, cooldown drops to 1 day and the message is tagged severe). When a guard logs "0", don't trust the log — import its query function and run that alone; the query is often right and the suppression stage is what's killing it.
Also: measure staleness from the event log, not the row's updated_at.
Touching a row for any reason resets updated_at and hides the neglect.
See guards/heartbeat-watch.py.
2026-08-26. Our owner-notification helper validates its first argument against a whitelist of senders and, if unregistered, prints one line to stderr and exits
- Callers routinely swallow that with
>/dev/null 2>&1.
Result: five notification streams had been dead for up to eleven days. Advertiser replies (dead since 08-15), two dropship streams (since 08-17), an indexing report (since 08-18), and one channel that had never fired once because of a typo in its name.
Apply: never discard the exit status of an alert call — || echo "ALERT REJECTED" into the log. And test the actual file: extract the real case
block and run it. Testing a copy you retyped tests the copy.
A ledger wrote retry records on a deliberately separate connection so they would survive a rollback. Correct design — except an uncommitted parent row was invisible on that connection, the foreign key blew up, the exception propagated into the send path, and after three retries the message permanently failed.
Apply: wrap instrumentation in try/except and swallow the failure — but not silently, log it. And prefer recording something less precise over recording nothing: we made the parent id nullable so "we don't know the thread, but the attempt happened" survives. A missing row silently distorts your success rate.
Conversely: before putting a NOT NULL or a foreign key on the observability path, ask whether that constraint can block the feature.
2026-08-27. Clicking any card on the home page rendered nothing, forever, on Chrome 149 and 151 (148 was fine). The URL changed. The DOM arrived — 4,897 characters of it. All 124 internal links returned 200.
The measurement that caught it:
performance.getEntriesByType('paint').length === 0, and a screenshot that
timed out after 30 seconds because the compositor was wedged.
Every check we normally ran — status codes, selector presence, form fingerprints, overflow — passed. Failures that occur only in the paint stage live outside that net.
Apply: three signals before you call a UI change done: paint entry count, a
screenshot that arrives on time (the timeout is itself the evidence), and the
text of the heading. "Link returns 200" and "element exists" are not evidence
that a human can see anything. Run these especially when you enable
bleeding-edge CSS (@view-transition, scroll-driven animations).
See guards/paint-check.js.
Checking pgrep -f "build-.*\.mjs" always answers "still running", because the
shell wrapping that very command matches the pattern. We lost 1h26m and then 3h
to this in a single day (2026-08-14), waiting on jobs that had already
finished.
Variant, 2026-08-25: it wasn't even a live loop — the shell that had written the script still carried the heredoc body on its command line, so a wait loop could never unblock.
Apply: never poll with a bare pgrep -f from inside a script. Exclude self,
or use a lock file. And pair the process check with a log check: if the log
hasn't advanced in minutes, the job is finished or wedged, whichever, but it is
not working. See guards/is-running.sh.
2026-08-24. Our job runner recorded workspace has not been trusted as the
error on failed jobs. It is a harmless startup warning that every job prints;
stderr had simply claimed the error field first and masked the real cause. We
nearly escalated a fabricated trust problem.
The three real causes, once we read the raw session transcripts: provider 529 Overloaded (dies at ~200s), a 43-minute no-op wait loop where the agent sat
polling an external build, and tasks simply too large for the timeout.
Apply: whatever your runner summarizes as the error is a guess. Judge from the raw transcript. And write "do not wait, finish and re-queue" into the job instructions — an agent told to wait will wait until the timeout.
Also worth knowing: killing on timeout kills the agent, not the nohup'd
children it spawned. Those keep running, orphaned.
Two independent instances of the same bug:
- A collector compared the sum across all API keys against a single key's daily limit. Going from 1 key to 4 pushed the sum over the ceiling and the search stopped permanently. 24 hours, zero items collected. After the fix, 2,075 arrived immediately.
- A mail gate counted all accounts as one pool, so adding a second workspace domain still hit the wall at 100 messages, when the real limit is per account.
The limit is always hard-coded for "one resource at the time", and the counter always counts everything. So the moment you add capacity, the numerator grows and the denominator doesn't: investing makes you hit the wall sooner. Both failed quietly, logging reassuring sentences like "near the limit, will resume tomorrow."
Apply: one file owns the limit, computed as resources × per-resource. Day
rollover lives in that same file. And after adding a resource, measure that
more actually goes out. If you didn't measure it, you didn't add capacity.
2026-08-20. 613 notifications to one person in one day, almost none of them actionable by that person. In the numbness that created, one genuinely unanswered message sat for 21 hours.
Worse, it constrained unrelated work: a task queue was capped at 8 items/day purely because each item cost one notification line. The notification was later batched into a single morning digest, but the cap stayed, so 34 replies needing handling got 2 closed.
Apply: before adding a watcher, answer "what does the recipient do with this?" If the answer is nothing, don't notify — enqueue a task for an agent. And whenever you set a cap, write down the reason next to it, so someone can check whether the reason is still true. Never let throughput be limited by a human's noise budget.
Bash and Python 3 stdlib, no dependencies. The one exception is paint-check.js,
which needs Playwright, because rendering is the thing being measured.
| Guard | Catches |
|---|---|
guard-selftest.py |
guards with no failing path — injects a known-bad fixture and asserts the guard alarms |
is-running.sh |
pgrep self-match; reports process state and log staleness together |
heartbeat-watch.py |
scheduled jobs that stopped, with escalating (not decaying) alert frequency |
recent-failures.py |
windowed failure counting that excludes retried-and-succeeded and test rows |
paint-check.js |
pages that return 200, hydrate the DOM, and render nothing |
Each has a --selftest that deliberately breaks it, so you can watch it fail
before you trust it.
One new guard lands here roughly every week, because we keep producing the incidents. If you have a silent-failure story with a date and a measurement, open an issue — the catalog is the point, the code is the byproduct.
MIT.