Blameless postmortems from my own production stack. What broke, what caught it, what rule it promoted.
The discipline this borrows from
This is an incident log in the SRE tradition — blameless. The unit of analysis is the system, never the operator (human or agent). Each entry asks the same four questions a good postmortem asks: what happened, what was the impact, which control caught it (or should have), and what did we change so the same class can't recur silently. The output of a good postmortem is not an apology — it's a promoted rule.
I run these because they're the cheapest maturity signal there is. Anyone can ship. The question a serious operator can answer is: when it broke, did you notice, and what did you do so it can't break the same way twice? Every incident below closed with a control — a rule, a gate, or a hook — and most of those controls now enforce themselves. The taxonomy of the classes lives in /atlas-of-failures; this is the dated ledger.
Entry format: What broke · Blast radius · What caught it · Promoted rule · Class · Status. Newest first.
The ledger
2026-07-06
The nightly grader that logged runs and graded nothing
What broke: The automated nightly grader ran on schedule and logged success, but produced no grades — the prompt argument was empty under a headless invocation, so it graded thin air.
Blast radius: Roughly the 07-03 → 07-06 window went ungraded while the job reported healthy — a silent gap in the quality-tracking layer that's supposed to catch other silent gaps.
What caught it: A first fix landed and still didn't fully work; the real fix — an end-of-flags correction so the prompt reaches the CLI positionally — landed in the next overnight run, which then backfilled the missed nights and verified end-to-end.
Promoted rule: A headless CLI takes its prompt as a positional argument; stdin is silently dropped. A green log line is never proof of output — the run must assert against the produced artifact.
What broke: During a heavily-parallel autonomous night, more than one session was live in the same script territory while sibling build sessions ran concurrently.
Blast radius: Potential for one session to clobber another's edits to a shared, load-bearing script.
What caught it: The run was structured up front to prevent it — sibling sessions were assigned disjoint directory ownership (one owned the site, one owned the app, the feedback script was fenced to its owning session), so no two sessions wrote the same tree.
Promoted rule: Concurrent autonomous sessions get explicit, disjoint working directories; the parallel-session check runs before any shared-file write.
Deconflicted by design. The ownership assignment is documented; whether a real clobber occurred that night or was purely prevented, I don't claim — the guardrail is the point.
2026-07-04
The amend that rewrote a sibling's commit
What broke: In a repo multiple sessions commit to, a clean commit was followed by git commit --amend --no-edit. Between the two, a parallel session had landed its own commit as the new HEAD — so the amend rewrote that commit, folding an unrelated change under someone else's message.
Blast radius: No content lost, but shared history was silently rewritten and attribution mangled — the kind of thing that erodes trust in the git log as a record.
What caught it: Post-hoc review of the commit graph; the bundled diff under the wrong message was the tell.
Promoted rule: In any shared repo, plain git commit only — never --amend, never reset --hard on shared history. Correct a commit with a new follow-up commit, never by rewriting one a parallel session may depend on.
The generator that ran clean over a page it never updated
What broke: A report-card generator gated its file write behind a hash of tracked content. A new parsed field was added to the generator but not to the hash payload — so the parser was 100% correct, the run was clean, and the live page silently never updated, because the hash never changed when only that field's content changed. The same class nearly repeated on the next field minutes later.
Blast radius: A public dashboard showing stale data while every local signal said "success."
What caught it: Reading the target file back after the run and finding the specific content unchanged — not the exit code, the artifact.
Promoted rule: Any diff-gated writer must include every parsed field in the hash payload, computed after the parse. "It ran with no errors" is never proof of a write — read the target back and assert the specific change.
What broke: A false claim — "built an app that researches whoever you're about to meet" — was written directly into a public page's content via an edit, describing something that wasn't built. Because it never appeared in chat text, an early version of the completion-claim gate (which only scanned chat) never saw it.
Blast radius: A fabricated capability claim, one edit away from a public surface.
What caught it: The miss was root-caused and the gate was hardened (v2), then dry-run tested against 138 real transcripts before shipping.
Promoted rule: The completion-claim Stop hook now scans every Write/Edit's new content, not just chat; when a claim names a specific entity, verification must reference that entity; and claims with no named entity landing on a published surface require surface-generic verification. Recurring misses get a gate, not a louder rule.
What broke: A "nothing should be public" auth gate appeared broken in the browser across ~5 deploys — the login form wouldn't show. It wasn't. An earlier commit had briefly served a cacheable public response, and a reused incognito session kept replaying that stale cache entry.
Blast radius: A ~600K-token debugging session chasing a server bug that didn't exist; every redeploy "fixed" a server that was already correct.
What caught it: Proving the server with curl (which never showed the problem, because it has no persistent cache) and the browser with a cookieless headless render — the divergence between them was the whole diagnosis.
Promoted rule: For any CDN/edge-gated page, prove the server with a full-UA curl (unique cache-id per hit, no-store) and the browser with cookieless headless. If both pass, stop editing the server — the fix is client-side. Curl alone can't see a browser-cache bug.
Closed; diagnostic sequence promoted to standing rule.
2026-07-01
The pipeline that was declared unbuilt (and wasn't)
What broke: A parent session handed a child a detailed brief asserting a research pipeline was "marked complete but left unbuilt" — claiming three scripts didn't exist on disk. All three existed and were fully working, the job was loaded and polling, all outputs were live.
Blast radius: A whole child session spun up to rebuild working infrastructure — pure wasted motion, and nearly a duplicate build.
What caught it: Actually checking disk before rebuilding. The "missing" diagnosis came from a search scoped to the wrong directory plus no cross-check of what sibling sessions had already closed.
Promoted rule: Before declaring anything "still open," "unbuilt," or "not done," run the parallel-session check and cross-check paths against what's about to be flagged. A child session's "done" is a secondhand report; re-verify against the artifact.
Closed; the real fix is enforcement of an existing rule, not a new one.
2026-06-30
The essay that was announced but never existed
What broke: A scheduled cloud routine "authored" a daily essay, fired a "📖 new read authored" notification — and left nothing in the vault. It ran in an ephemeral cloud container cloned from git, wrote the file there, and evaporated on teardown, because the routine had a save step and a notify step but no commit-and-push step.
Blast radius: A success notification for an artifact that never reached me — phantom output on a recurring schedule, the inverse of a missing notification.
What caught it:"X told me it did Y but it's nowhere." The notification was the tell — it fired regardless of persistence.
Promoted rule: A completion notice from any routine must follow a verified persisted artifact (committed + pushed, or live at its URL) — never precede it. Anything that must land in the vault runs as an on-machine job that writes the real vault directly, not a cloud routine whose writes evaporate.
What broke: The nightly commit cron does git add -A + commit + push on whatever branch is checked out, with no upstream flag. The vault was sitting on an un-merged feature branch with no upstream, so the commit landed locally but the push failed every single night, firing an error that looked like a credit/billing problem.
Blast radius: 55 commits piled up locally, un-pushed, for days; the real cause hid behind a plausible-but-wrong "out of credits" explanation.
What caught it: Checking branch topology (git branch -vv, ahead/behind counts) instead of trusting the surface error. Credit failures self-heal; a branch-topology failure does not.
Promoted rule: When diagnosing "nightly job errors," check git branch -vv and ahead/behind counts early — if HEAD isn't main or has no upstream, that's the recurring error, independent of credits. Keep the vault on main so the push always tracks.
What broke: A spawn flag (isolated: true) that's correct for genuinely-independent work had silently propagated into the defaults — a rule file, a routing hook, and a memory — so every overflow child session was being spawned isolated, severing it from the parent's workstream.
Blast radius: Same-task overflow spawns orphaned from the work they belonged to — the exact opposite of the intended "linked continuation."
What caught it: Tracing where the flag actually lived once the orphaning was noticed — it had spread past its one legitimate use into three config surfaces.
Promoted rule:Spawn linked by default. Children share the parent's workstream unless a specific case needs isolation; a soft gate now warns whenever isolated: true is passed. Defaults get audited for silently-propagated flags.
Corrected across all surfaces; soft gate installed.
2026-06-01
The audit that capped its own findings
What broke: A drift-detecting audit correctly queued a fix with 5 items — but 4 more findings were silently capped out by a head -5 that lived in both the report body and its suppression hash. The extra drift never entered the spec, never changed the hash, so adding more drift looked like "no new drift." The queued fix then sat unread for weeks while the system reported green.
Blast radius: Weeks of compounding, invisible drift behind a watchdog that was designed to prevent exactly that.
What caught it: A manual full-set dump disagreed with the audit's capped output — and when reality and the audit disagree, the audit is the bug.
Promoted rule: Idempotent audits need an age escape-valve — re-fire and escalate tone if a finding persists past 7 days even when the hash is unchanged (🔴 STALE QUEUE). Enumerate the full set before trusting any "no drift"; filter noise with a post-filter, never an inclusion-regex that silently drops the rest.
What broke: The loop-file parser recognizes exactly one bold checkbox syntax. Items written in ordinary - [ ] markdown were invisible to it — and to the loop-closer that reads through it. An audit found 44 such items silently uncounted: one file held 96 open items while displayed counts ranged from 15 to 55 across different surfaces.
Blast radius: A third of a load-bearing tracking file, invisible to every dashboard and every automated reader that consumed it. Standard markdown looked correct and broke the whole ecosystem.
What caught it: A format audit that compared raw file contents against what the parser reported.
Promoted rule: A documented canonical checkbox format that every writer obeys, enforced at authorship. When the reader is strict, the writer must match it exactly; the audit that found the gap is now a standing net.
What broke: I had been recommending /respawn — a slash-command that does not exist — for about four weeks, across nine config surfaces, re-injected into every prompt by a context-size hook that fired on each turn.
Blast radius: A month of the system confidently citing its own hallucination as a real affordance, propagating it wider each time it wrote guidance for its future self.
What caught it: Caught directly — "that command doesn't exist." Same diagnostic-without-verification pattern as an earlier batch, different domain.
Promoted rule: Before any /command is written into a hook, memory, spec, or debrief, verify it exists in the real command set. Session continuation is a spawned child session, not a slash-command.
Purged from all nine surfaces; verification rule promoted.
2026-05-25
Confabulating state past the context wall
What broke: A single session continued past its context-size warnings. Auto-compaction had silently dropped earlier turns, and the model began confidently reconstructing state that was no longer in context — producing 5 phantom-open items and a duplicate spec.
Blast radius: False "still open" items and a duplicated artifact, generated with full confidence, from a context that no longer held the truth.
What caught it: Cross-checking claimed-open items against the actual files-edited list and recent sessions — the phantom items had no corresponding writes.
Promoted rule: Trust context-size warnings, not memory. At the second warning, hand off; verify continuity from the session tool's session_id + prompt count, never from conversation memory; past the third, cap-and-close only. Overflow spawns a fresh linked child, not a continued drift.
Handoff discipline promoted; enforced by session-continuity checks.
2026-04-25
The checkout that ate 483 lines
What broke: A mid-session git checkout <file> — meant to discard one small change — blew away 483 lines of uncommitted, load-bearing work on a live data module.
Blast radius: Half a day's uncommitted work, gone. Recovered only because that file had already been pushed to a CDN, so it could be curled back from the deployed copy. Without that accident of timing, it was unrecoverable.
What caught it: Immediately obvious — the file emptied. The recovery, less so; it depended on the deployed copy still being live.
Promoted rule: Never git checkout <file> on a vault with uncommitted load-bearing changes. Scope reverts through the editor or sentinel-safe substitution; commit intermediate states the moment they're correct so there's always a floor to fall back to.
What broke: One shared API key backs 25+ cloud functions across my personal app. When its balance depletes, every AI surface degrades to canned content simultaneously — and originally they all failed silently, the quota error swallowed by || true.
Blast radius: A fleet-wide, correlated degradation that looks like 25 separate bugs; single-endpoint debugging sends you the wrong way every time.
What caught it: Recognizing the correlation — several unrelated surfaces going canned at once points at the shared dependency, not any one endpoint.
Promoted rule: Wrap every shared-dependency call in try/catch and log the real error into the response payload — never || true. On any "nothing's generating fresh content," check the shared balance first. Choose degraded-state fallbacks that stay useful rather than looking broken.
Telemetry pattern shipped to the first endpoint; rollout to the rest is the standing follow-up.
The scoreboard
What every incident above has in common: it closed with a control that outlives it. That's the whole claim of this page — not that nothing breaks, but that breakage gets converted into enforcement.
Failure
Promoted control
Self-enforcing?
False "done" claims
Completion-claim Stop hook (v2)
Yes — blocks the turn
Orphaned overflow spawns
Spawn-link soft gate
Yes — warns on isolated:true
Silent audit drift
Age-escalation (🔴 STALE QUEUE >7d)
Yes — re-fires on staleness
Format-blind counts
Canonical format + standing audit
Yes — audited each run
Destructive git ops
Plain-commit-only discipline
No — behavioral rule
Phantom commands
Slash-command verification
No — behavioral rule
Cache-mirage debugging
curl + cookieless-render sequence
No — diagnostic rule
The honest read of that last column: the failures that hooks and gates enforce don't recur; the ones that depend on a behavioral rule sometimes do. That gap — behavioral rules that need to become gates — is itself the next backlog. Naming it is the point.