then round again
The system is my lab: AI jobs that run on a schedule. In one set of them, about one run in four wrote almost nothing, and every status screen still said success: 47 out of 47.
… every status surface reports those runs as `success` — history.json is Counter({'success': 47}), 47 of 47, including a 38 B run and a `529 Overloaded` run. A field that cannot say no is not a check.Its code, lines 6 to 9
From the notes at the top of the control’s code.
A control is a check that runs by itself, outside the AI. This one reads what each run wrote. Under 400 bytes, about a short paragraph, means hollow: it said success and wrote almost nothing. Empty runs wrote 266 at most; the smallest working one, 593.
hollow run -> file exists, 0-266 B (measured)
non-run -> no file at all
healthy run -> file exists, 593 B - 10 KB (measured; …)Its code, lines 14 to 16
From the control’s own notes.
The control is a scheduled job too, so each morning a dashboard marks it like the other 63: green if it ran on time with nothing wrong.
How the system is graded
cleanneeds a look: late, or it raised somethingfailedno reportswitched offshare of a group clean that morning
Yellow means look, not broken: on 2 October, 9 of 10 yellow jobs had run on schedule.
16 personal jobs are counted; names stay private.
Every control has a way around it. Size alone would miss a run that fails late. One in July wrote 2,169 bytes, then quit on an error from the AI service. So the control also looks for the error message, at any size.
[2026-09-09 16:25] HOLLOW|capture-close-backfill|2026-07-16-capture-close-backfill.md|2169B|api-errorIts log, line 43
From its first scan, the day it was built.
Where it keeps what it learns is in the lab: How the AI remembers →
My research agent’s essays were being scored by a script that counts words, headers and dashes. After a blind test came back, Claude said it had killed the metric. I didn’t follow, so I asked:
what do you mean, killed the metric?
16 September 2026
Claude explains on the spot. Terms worth keeping go on a list, and the ones I approve become entries in my glossary, each with where it came up. It holds 111 terms. This is that entry, cut into cards.
“what do you mean, killed the metric?”
My real glossary. This entry is on the next cards.
Killed means thrown out: the score wasn’t measuring what we thought. A script rated how different a set of essays sounded by counting five things, like words and dashes. Every count was right. Each was only a stand-in, a proxy, for the real question: does a reader hear different writers?
Decide whether two people look alike by comparing height, weight, shoe size and hair length. Every measurement correct, repeatable, cheap. None of them measures the face.
Construct validity asks whether a measure captures the thing it claims to measure. That is a separate question from whether it is reliable (the same answer twice) or computed right. A metric can be flawless on both and empty on this one.
I was the judge. I couldn’t see which essay was which, but by the fourth pass I knew what the study was asking. If your judge can guess the result you are hoping for, the judge is a stand-in too, and nothing real is underneath.
Ask of any AI score: what is this number a stand-in for, and what did you check it against? Then add one case you predict will fail, and see whether it does. A study that can only confirm is not a study.
Just after midnight each night, a scheduled AI job reads what I did with AI that day and grades it like school, A to F. A failed run retries four times that day. Every day in September has a grade.
Each day gets up to nine letters: eight for things like how clear my prompts were, what it cost and whether I wrote a plan first, then one overall. On the real card each letter opens to why, and what to do differently. Nobody independent grades the grader yet.
