Blameless postmortems from my own production stack. What broke, what caught it, what rule it promoted.
The discipline this borrows from
This is an incident log in the SRE tradition — blameless. The unit of analysis is the system, never the operator (human or agent). Each entry asks the same four questions a good postmortem asks: what happened, what was the impact, which control caught it (or should have), and what did we change so the same class can't recur silently. The output of a good postmortem is not an apology — it's a promoted rule.
I run these because they're the cheapest maturity signal there is. Anyone can ship. The question a serious operator can answer is: when it broke, did you notice, and what did you do so it can't break the same way twice? Every incident below closed with a control — a rule, a gate, or a hook — and most of those controls now enforce themselves. The taxonomy of the classes lives in /atlas-of-failures; this is the dated ledger.
Entry format: What broke · Blast radius · What caught it · Promoted rule · Class · Status. Newest first.
The ledger
2026-08-31
Verified the numbers, shipped the prose
What broke: A specialist subagent's output was integrated into a private research artifact. Every numeric claim it made was reconciled against the source data and confirmed correct. Its prose was relayed verbatim — and carried an unverified historical superlative presented as established fact.
Blast radius: One unsupported claim in a private reference file. The larger cost was the verification note itself, which implied whole-document coverage while covering only the arithmetic — the artifact looked more checked than it was.
What caught it:Not me.test-claim-enforcement-gate.sh, a Stop hook that blocks a completion claim naming a specific entity when nothing in the turn actually verified that entity. It fired on the named claim, not on the numbers — which is exactly the gap the numbers had hidden.
Promoted rule: Verification scope is a boundary to be stated, not a checkmark. An artifact mixing checked and unchecked material must name the unchecked categories by name, in the artifact. Rigor on the checkable part does not extend to the rest, and reads as though it does.
Corrected in the artifact. The unverified claim was checked and narrowed; the unchecked categories are now enumerated on the file by name.
2026-08-31
The parser that returned a believable zero
What broke: A separator character class in a coordinate parser consumed the sign character of negative values, so every parsed value silently changed sign. The query completed and exited 0 with an empty result set — and empty was a defensible answer, so nothing raised.
Blast radius: None shipped. Had it survived, the analysis would have been inverted rather than failed — the worst available outcome, because an inverted result is reviewable and a crash is not.
What caught it: A pre-build spec audit that ran the parser against a pinned known-good value instead of reasoning about the pattern, and hit the sign flip live. The same run found two of four public API mirrors returning empty bodies, which would have been read as "no candidates exist" rather than "the fetch failed."
Promoted rule: A parser fixture asserts the round-trip of a signed value, not that the parser runs. Every population query carries a floor assertion, and zero is never accepted without a positive control proving the query can return non-zero. An empty upstream response aborts; it never fails open into an empty set.
Caught pre-build. No corrupted output was produced. No persisted control yet — the fixture discipline is per-build.
2026-08-31
The subtraction that returned the city
What broke: A complement analysis subtracted an observed set from a much larger candidate set to find what was absent. The observed set was too small, relative to the candidates, for the subtraction to remove a meaningful fraction — so the "finding" was the candidate set with a rounding error taken out.
Blast radius: None shipped, but this is the failure that would have survived review most easily. Nothing was incorrect. No assertion could have failed. The output had the exact shape of a result and was about nothing.
What caught it: A pre-build audit that measured the two set sizes against each other before any code ran, rather than reviewing the method as prose. The ratio made the problem immediate; the prose description had read as perfectly sound.
Promoted rule: Before building a difference, compute what fraction the subtrahend removes. Single-digit percent means the unit of analysis is wrong — change the unit until the operation can discriminate, and do it before the build, not in review of the output.
Method changed before build. Unit of analysis moved from point to area; the rebuilt analysis discriminated.
2026-07-06
The nightly grader that logged runs and graded nothing
What broke: The automated nightly grader ran on schedule and logged success, but produced no grades — the prompt argument was empty under a headless invocation, so it graded thin air.
Blast radius: Roughly the 07-03 → 07-06 window went ungraded while the job reported healthy — a silent gap in the quality-tracking layer that's supposed to catch other silent gaps.
What caught it: A first fix landed and still didn't fully work; the real fix — an end-of-flags correction so the prompt reaches the CLI positionally — landed in the next overnight run, which then backfilled the missed nights and verified end-to-end.
Promoted rule: A headless CLI takes its prompt as a positional argument; stdin is silently dropped. A green log line is never proof of output — the run must assert against the produced artifact.
What broke: During a heavily-parallel autonomous night, more than one session was live in the same script territory while sibling build sessions ran concurrently.
Blast radius: Potential for one session to clobber another's edits to a shared, load-bearing script.
What caught it: The run was structured up front to prevent it — sibling sessions were assigned disjoint directory ownership (one owned the site, one owned the app, the feedback script was fenced to its owning session), so no two sessions wrote the same tree.
Promoted rule: Concurrent autonomous sessions get explicit, disjoint working directories; the parallel-session check runs before any shared-file write.
Deconflicted by design. The ownership assignment is documented; whether a real clobber occurred that night or was purely prevented, I don't claim — the guardrail is the point.
2026-07-04
The amend that rewrote a sibling's commit
What broke: In a repo multiple sessions commit to, a clean commit was followed by git commit --amend --no-edit. Between the two, a parallel session had landed its own commit as the new HEAD — so the amend rewrote that commit, folding an unrelated change under someone else's message.
Blast radius: No content lost, but shared history was silently rewritten and attribution mangled — the kind of thing that erodes trust in the git log as a record.
What caught it: Post-hoc review of the commit graph; the bundled diff under the wrong message was the tell.
Promoted rule: In any shared repo, plain git commit only — never --amend, never reset --hard on shared history. Correct a commit with a new follow-up commit, never by rewriting one a parallel session may depend on.
The generator that ran clean over a page it never updated
What broke: A report-card generator gated its file write behind a hash of tracked content. A new parsed field was added to the generator but not to the hash payload — so the parser was 100% correct, the run was clean, and the live page silently never updated, because the hash never changed when only that field's content changed. The same class nearly repeated on the next field minutes later.
Blast radius: A public dashboard showing stale data while every local signal said "success."
What caught it: Reading the target file back after the run and finding the specific content unchanged — not the exit code, the artifact.
Promoted rule: Any diff-gated writer must include every parsed field in the hash payload, computed after the parse. "It ran with no errors" is never proof of a write — read the target back and assert the specific change.
What broke: A false claim — "built an app that researches whoever you're about to meet" — was written directly into a public page's content via an edit, describing something that wasn't built. Because it never appeared in chat text, an early version of the completion-claim gate (which only scanned chat) never saw it.
Blast radius: A fabricated capability claim, one edit away from a public surface.
What caught it: The miss was root-caused and the gate was hardened (v2), then dry-run tested against 138 real transcripts before shipping.
Promoted rule: The completion-claim Stop hook now scans every Write/Edit's new content, not just chat; when a claim names a specific entity, verification must reference that entity; and claims with no named entity landing on a published surface require surface-generic verification. Recurring misses get a gate, not a louder rule.
What broke: A "nothing should be public" auth gate appeared broken in the browser across ~5 deploys — the login form wouldn't show. It wasn't. An earlier commit had briefly served a cacheable public response, and a reused incognito session kept replaying that stale cache entry.
Blast radius: A ~600K-token debugging session chasing a server bug that didn't exist; every redeploy "fixed" a server that was already correct.
What caught it: Proving the server with curl (which never showed the problem, because it has no persistent cache) and the browser with a cookieless headless render — the divergence between them was the whole diagnosis.
Promoted rule: For any CDN/edge-gated page, prove the server with a full-UA curl (unique cache-id per hit, no-store) and the browser with cookieless headless. If both pass, stop editing the server — the fix is client-side. Curl alone can't see a browser-cache bug.
Closed; diagnostic sequence promoted to standing rule.
2026-07-01
The pipeline that was declared unbuilt (and wasn't)
What broke: A parent session handed a child a detailed brief asserting a research pipeline was "marked complete but left unbuilt" — claiming three scripts didn't exist on disk. All three existed and were fully working, the job was loaded and polling, all outputs were live.
Blast radius: A whole child session spun up to rebuild working infrastructure — pure wasted motion, and nearly a duplicate build.
What caught it: Actually checking disk before rebuilding. The "missing" diagnosis came from a search scoped to the wrong directory plus no cross-check of what sibling sessions had already closed.
Promoted rule: Before declaring anything "still open," "unbuilt," or "not done," run the parallel-session check and cross-check paths against what's about to be flagged. A child session's "done" is a secondhand report; re-verify against the artifact.
Closed; the real fix is enforcement of an existing rule, not a new one.
2026-06-30
The essay that was announced but never existed
What broke: A scheduled cloud routine "authored" a daily essay, fired a "📖 new read authored" notification — and left nothing in the vault. It ran in an ephemeral cloud container cloned from git, wrote the file there, and evaporated on teardown, because the routine had a save step and a notify step but no commit-and-push step.
Blast radius: A success notification for an artifact that never reached me — phantom output on a recurring schedule, the inverse of a missing notification.
What caught it:"X told me it did Y but it's nowhere." The notification was the tell — it fired regardless of persistence.
Promoted rule: A completion notice from any routine must follow a verified persisted artifact (committed + pushed, or live at its URL) — never precede it. Anything that must land in the vault runs as an on-machine job that writes the real vault directly, not a cloud routine whose writes evaporate.
What broke: The nightly commit cron does git add -A + commit + push on whatever branch is checked out, with no upstream flag. The vault was sitting on an un-merged feature branch with no upstream, so the commit landed locally but the push failed every single night, firing an error that looked like a credit/billing problem.
Blast radius: 55 commits piled up locally, un-pushed, for days; the real cause hid behind a plausible-but-wrong "out of credits" explanation.
What caught it: Checking branch topology (git branch -vv, ahead/behind counts) instead of trusting the surface error. Credit failures self-heal; a branch-topology failure does not.
Promoted rule: When diagnosing "nightly job errors," check git branch -vv and ahead/behind counts early — if HEAD isn't main or has no upstream, that's the recurring error, independent of credits. Keep the vault on main so the push always tracks.
What broke: A spawn flag (isolated: true) that's correct for genuinely-independent work had silently propagated into the defaults — a rule file, a routing hook, and a memory — so every overflow child session was being spawned isolated, severing it from the parent's workstream.
Blast radius: Same-task overflow spawns orphaned from the work they belonged to — the exact opposite of the intended "linked continuation."
What caught it: Tracing where the flag actually lived once the orphaning was noticed — it had spread past its one legitimate use into three config surfaces.
Promoted rule:Spawn linked by default. Children share the parent's workstream unless a specific case needs isolation; a soft gate now warns whenever isolated: true is passed. Defaults get audited for silently-propagated flags.
Corrected across all surfaces; soft gate installed.
2026-06-01
The audit that capped its own findings
What broke: A drift-detecting audit correctly queued a fix with 5 items — but 4 more findings were silently capped out by a head -5 that lived in both the report body and its suppression hash. The extra drift never entered the spec, never changed the hash, so adding more drift looked like "no new drift." The queued fix then sat unread for weeks while the system reported green.
Blast radius: Weeks of compounding, invisible drift behind a watchdog that was designed to prevent exactly that.
What caught it: A manual full-set dump disagreed with the audit's capped output — and when reality and the audit disagree, the audit is the bug.
Promoted rule: Idempotent audits need an age escape-valve — re-fire and escalate tone if a finding persists past 7 days even when the hash is unchanged (🔴 STALE QUEUE). Enumerate the full set before trusting any "no drift"; filter noise with a post-filter, never an inclusion-regex that silently drops the rest.
What broke: The loop-file parser recognizes exactly one bold checkbox syntax. Items written in ordinary - [ ] markdown were invisible to it — and to the loop-closer that reads through it. An audit found 44 such items silently uncounted: one file held 96 open items while displayed counts ranged from 15 to 55 across different surfaces.
Blast radius: A third of a load-bearing tracking file, invisible to every dashboard and every automated reader that consumed it. Standard markdown looked correct and broke the whole ecosystem.
What caught it: A format audit that compared raw file contents against what the parser reported.
Promoted rule: A documented canonical checkbox format that every writer obeys, enforced at authorship. When the reader is strict, the writer must match it exactly; the audit that found the gap is now a standing net.
What broke: I had been recommending /respawn — a slash-command that does not exist — for about four weeks, across nine config surfaces, re-injected into every prompt by a context-size hook that fired on each turn.
Blast radius: A month of the system confidently citing its own hallucination as a real affordance, propagating it wider each time it wrote guidance for its future self.
What caught it: Caught directly — "that command doesn't exist." Same diagnostic-without-verification pattern as an earlier batch, different domain.
Promoted rule: Before any /command is written into a hook, memory, spec, or debrief, verify it exists in the real command set. Session continuation is a spawned child session, not a slash-command.
Purged from all nine surfaces; verification rule promoted.
2026-05-25
Confabulating state past the context wall
What broke: A single session continued past its context-size warnings. Auto-compaction had silently dropped earlier turns, and the model began confidently reconstructing state that was no longer in context — producing 5 phantom-open items and a duplicate spec.
Blast radius: False "still open" items and a duplicated artifact, generated with full confidence, from a context that no longer held the truth.
What caught it: Cross-checking claimed-open items against the actual files-edited list and recent sessions — the phantom items had no corresponding writes.
Promoted rule: Trust context-size warnings, not memory. At the second warning, hand off; verify continuity from the session tool's session_id + prompt count, never from conversation memory; past the third, cap-and-close only. Overflow spawns a fresh linked child, not a continued drift.
Handoff discipline promoted; enforced by session-continuity checks.
2026-04-25
The checkout that ate 483 lines
What broke: A mid-session git checkout <file> — meant to discard one small change — blew away 483 lines of uncommitted, load-bearing work on a live data module.
Blast radius: Half a day's uncommitted work, gone. Recovered only because that file had already been pushed to a CDN, so it could be curled back from the deployed copy. Without that accident of timing, it was unrecoverable.
What caught it: Immediately obvious — the file emptied. The recovery, less so; it depended on the deployed copy still being live.
Promoted rule: Never git checkout <file> on a vault with uncommitted load-bearing changes. Scope reverts through the editor or sentinel-safe substitution; commit intermediate states the moment they're correct so there's always a floor to fall back to.
What broke: One shared API key backs 25+ cloud functions across my personal app. When its balance depletes, every AI surface degrades to canned content simultaneously — and originally they all failed silently, the quota error swallowed by || true.
Blast radius: A fleet-wide, correlated degradation that looks like 25 separate bugs; single-endpoint debugging sends you the wrong way every time.
What caught it: Recognizing the correlation — several unrelated surfaces going canned at once points at the shared dependency, not any one endpoint.
Promoted rule: Wrap every shared-dependency call in try/catch and log the real error into the response payload — never || true. On any "nothing's generating fresh content," check the shared balance first. Choose degraded-state fallbacks that stay useful rather than looking broken.
Telemetry pattern shipped to the first endpoint; rollout to the rest is the standing follow-up.
The scoreboard
What every incident above has in common: it closed with a control that outlives it. That's the whole claim of this page — not that nothing breaks, but that breakage gets converted into enforcement.
Failure
Promoted control
Self-enforcing?
False "done" claims
Completion-claim Stop hook (v2)
Yes — blocks the turn
Orphaned overflow spawns
Spawn-link soft gate
Yes — warns on isolated:true
Silent audit drift
Age-escalation (🔴 STALE QUEUE >7d)
Yes — re-fires on staleness
Format-blind counts
Canonical format + standing audit
Yes — audited each run
Destructive git ops
Plain-commit-only discipline
No — behavioral rule
Phantom commands
Slash-command verification
No — behavioral rule
Cache-mirage debugging
curl + cookieless-render sequence
No — diagnostic rule
The honest read of that last column: the failures that hooks and gates enforce don't recur; the ones that depend on a behavioral rule sometimes do. That gap — behavioral rules that need to become gates — is itself the next backlog. Naming it is the point.