Evals Are a Supply Chain Problem

Where the golden rule comes from

Small vendors in consumer goods live inside metrics they cannot see the inside of. A vendor scorecard says GREEN or YELLOW, but the threshold behind the color and the population you were compared against are not published. A velocity verdict arrives without its baseline. Nobody is being malicious: category data is proprietary, comparisons belong to whoever compiled them, and the honest buyers quietly share what context they can. But the structural effect is a decade of being measured by instruments you cannot audit, and my golden rule came out of it: pin the definitions before you do anything. A verdict without its baseline attached is not a measurement; it is a verdict wearing a measurement's clothes.

Terms, before anything else. An evaluation (an “eval”) is a battery of tests that measures what an AI model can and cannot do. An eval pipeline is the factory that runs those tests continuously, at scale, while the model is still being trained. That factory is the machine that either repeats the secret-scorecard sin at industrial scale or refuses to. This essay walks the whole pipeline, and because I learned its concepts by mapping them onto the operation I actually ran, it is written in that dialect. The mapping is not decoration. It is lossless: every design decision in a modern eval pipeline has a loading-dock ancestor, and program managers who ran physical operations already know this job. They just know it under other names.

The dictionary, up front

(A BOL is a bill of lading: the shipping document that lists exactly what is on the truck. It will come up.)

The order: checkpoints, pallets, and affordable damage

Training never stops. Every so often it emits a checkpoint: that frozen snapshot of the model as it exists that hour: and the checkpoint is a purchase order: a full battery of evaluations must run against it, on a deadline, while the next PO is already forming behind it.

You do not hand one worker the whole order, and you do not cut it into ten thousand pieces of paperwork either. You build pallets: the work split into chunks (software calls them shards), sized by one rule, how much damage you can afford to eat when one falls off the forklift. A dropped shard should be an annoyance to rebuild, never a catastrophe, and the paperwork per pallet should never cost more than the picking. Every sharding decision in distributed work is this tradeoff, and operations people have priced it forever.

The whiteboard, not the conveyor

The order board is a program manager's oldest artifact: one place that answers “what is still open, who has it, what died” instantly. The rookie move is buying the Amazon-scale conveyor system for whiteboard-scale volume. The veteran runs the whiteboard, because the real requirement was never throughput; it was that the pick list and the status board be the same document, so reconciliation is reading the board, not cross-checking three systems. In software terms that is a plain, boring database instead of an industrial message-passing system, and choosing boring is a program decision, not a technical one.

One mechanism makes the board self-service: a worker takes the topmost unclaimed line and initials it in one motion. Nobody can grab a line already initialed; nobody waits for a dispatcher. Priority is honored the dock way: perishables ship first, and today's perishables outrank yesterday's dry goods, because a fresh vital sign is worth more than a stale deep-dive.

One status per pallet

A claimed pallet is a lease, and the worker radios in every few minutes: a heartbeat. The office does not guess how long picking should take, because task duration varies wildly; it only tracks whether a healthy worker could still reach the radio, which is nearly constant. Miss enough calls and your initials get erased, and the pallet returns to the lane for the next worker.

But you tally the bounces, because a pallet that has come back three times is not unlucky; something is wrong with that pallet. Sending it out a fourth time is denial. It goes to the hold cage with a red tag stating why: not thrown away, not quietly shipped short. Caged, tagged, visible. Software calls this a dead-letter queue with bounded retries. The dock calls it Tuesday.

Declared shorts, or the honest bill of lading

The truck leaves when the board shows every pallet either Done or Dead, and the dead ones go on the face of the bill of lading: shipped 9,847 of 9,850, three in the hold cage, tags attached. The customer receives a short-ship they can act on instead of a mystery shortage they discover at receiving. A silent shortage becomes a claim, a chargeback, and trust damage. A declared shortage is just logistics.

Enforcement is physical, not procedural. The filing cabinet has one pre-labeled slot per pallet, cut when the order drops; a duplicate BOL has nowhere to go. Integrity by construction beats integrity by discipline: you can staff the dock with temps precisely because the cabinet is idiot-proof. The order closes on a three-way match, the same reconciliation I ran monthly against distributor claims: what was ordered, what was filed, what was graded, agreeing line by line, with any orphan line treated as an alarm. And you keep the raw paper: full transcripts, timings, versions: because summaries cannot be re-audited, and storage is the cheapest thing in this entire economy.

Pin the definitions

“Did the model solve the task” is exactly as undefined as “are your sales good” until someone writes the rule down. The scorer (the code that decides whether an answer counts as correct) is that rule made executable, which means grading rules are contracts under change control, not drafts: version stamped on every graded row, changes landing at declared effective dates, history restated rather than silently mixed. Change the rule without the stamp and your own grader manufactures a fake regression, indistinguishable from the model changing.

The same discipline decides what a number is allowed to claim. A launch demo (“the model built Minecraft!”) is one attempt, shown precisely because it worked (you never see the tries before it), scope undefined, quite possibly reciting the most-tutorialized game in history. A demo is a claim about the maximum; an eval is a claim about the distribution, and steering decisions ride on distributions. My decade of telling a buyer's anecdote apart from my own scan data transfers to this without modification.

The pager that gets believed

The enemy of monitoring is not the missed regression (a real drop in what the model can do); it is the alarm that cries wolf, because a paged human who finds nothing three times snoozes the fourth page, the real one. Every eval score is an estimate from a limited sample of test items, so it wiggles from run to run even when nothing changed, the way a poll wiggles between surveys. Alert on raw point drops and you burn the on-call team's trust on noise.

So the alarm threshold is set relative to each eval's own measured wiggle, the alarm watches for trends rather than points, correlated dips across many evals route the page to infrastructure rather than research, and the page arrives carrying its own briefing: band, trend, canary status (canaries: a few dead-stable reference tasks whose scores should never move; if they move, the pipeline itself is broken), completion rate, versions, links to the raw work. Pallet photos attached to the claim, not filed in another building.

I ran this triage protocol for years over the phone. A store's dashboard once told me my fill rate was 14%. Whose fault is that? First, audit your own house: my shipping docs, my inventory to the DC, cheapest evidence first. Clean. Next, test the far end: call the store. “Have you been ordering?” “Yeah, we ain't getting it.” The break is localized to the distributor's hop in one phone call. Route the finding to the owner of the broken segment with the evidence assembled, and then write the attribution back to the record, so the bad number next to your name does not quietly become your reputation. A correct measurement with wrong attribution is as dangerous as a wrong measurement.

Dock engineering: shims, and which way to hold the 2x4

Trailer with no pallet jack; forklift only reaches the door edge. The correct solutions (driver carries a jack, buy a reach truck, hand-stack the cases) are respectively not your decision, absurd, and labor you cannot spare. So: a sturdy piece of 2x4 against the pallet, pushed with the fork tips. Ten cents of lumber converts the machine you have into the machine you need. Software calls this a shim, and every real pipeline is load-bearing shims at the boundaries.

The craft is in the orientation. Wide side: more contact, forgiving of misalignment, and it splits under concentrated load, a gradual-then-sudden failure that was weakening for ten pushes before the one where it lets go. That is the lenient adapter that absorbs malformed input until something downstream snaps. Narrow side: stronger against the actual force, but if your fork is not aligned it slips immediately and visibly. That is the strict validator that fails loudly at the boundary on push one. Pick the orientation by which failure you can afford: for data crossing into the ledger, narrow side every time, because a slip you see beats a split you don't. And when the load justifies it, laminate: a tolerant layer facing the messy world, a merciless one guarding the cabinet.

Three rules keep the trick honest: a shim is legitimate, a shim gets inspected (prove it does not crush the cases before trusting it), and a 2x4 used every Tuesday is a process defect, a real fix that is overdue and quietly accruing interest.

The bench: same recipe, different kettles

Under everything sits serving: the computers that actually run the model to answer the test questions. Two co-packers running the same recipe produce slightly different product: different kettles, different heat curves. Two different chip types running the same model behave the same way: they do their arithmetic in slightly different orders, and once in a while that tips a borderline word choice. So an eval number is a property of the model and the hardware it ran on, and the standard procedure is the co-packer procedure verbatim: pin the plant for comparisons, record the plant on every lot, and qualify a new plant against the old one before any dashboard mixes them.

The economics: chips are scarce, storage is free

The entire cost structure of an eval platform follows from one price difference: the chips that run the model are the scarce, fought-over resource, and the storage that keeps records is nearly free. Every rule below is that one sentence, applied.

The last two rows are the organizational half, and they were my day job. The eval fleet lives inside a compute budget that someone will eventually eye, and the team that keeps its bench is the one holding receipts. I fought for shelf space with velocity data for a decade. Same fight, same weapon.

The through-line

The thing that wrecks an entire freight shipment is one wrong address on a bill of lading. I have received other companies' misrouted freight to prove it. The thing that manufactures a false research finding is one capital letter silently breaking a join. Tiny clerical defects, catastrophic blast radius, in both worlds, which is why in both worlds the byte-level check runs first.

None of this required inventing new concepts to cross over. Leases, idempotent ledgers, dead-letter cages, three-way reconciliation, definitions under change control, honest status reporting, capacity defended with receipts: operations program managers have run all of it, on docks and phone calls, against counterparties with worse logging than any software system. The dialect differs. The program is the same.