Fifteen Questions Before I Trust a Number

What an eval is, in one breath

An evaluation ("eval") is a structured test for an AI system: a set of tasks, a rule for scoring them, enough of them to mean something, and a clear claim about what the score is supposed to measure. Most of the trouble in this field comes from the last part. The tasks get built, the scoring gets automated, the number gets published, and nobody wrote down what it was a number of.

Every question below is a way of asking the same thing from a different side: does this score track the thing we say it tracks? Researchers call that construct validity. I call it pinning the definitions before you do anything, which is what eleven years of being graded by vendor scorecards I could not see the inside of taught me.

1What exactly is being measured?

One sentence, written down before the first run. If it takes three sentences, the eval measures three things and reports one number. This is the question the other fourteen hang off: every one of them is a way of checking whether the score still matches this sentence.

A measurement sentence has to name three things. The population: which tasks, from where, how many. The rule: what counts as a pass, decided by what, with no judgement call left to the reader. The scope of the claim: what the score is allowed to be called afterwards. Leave any of the three out and the number will be filled in by whoever reads it, usually in the most generous direction.

The left column is what gets published. The right column is what was tested. The gap between them is where every other question on this page lives.

A test for whether the sentence is doing its job: can two people disagree about the score without disagreeing about the sentence? If yes, the sentence is the contract and the argument is about data, which is the argument you want. If the disagreement is about what the sentence meant, there was no measurement, only a number.

Some things you want to measure really are several things. "Safe" or "useful" does not fit in one sentence, and forcing it there is its own kind of dishonesty. The rule survives: write one sentence per thing, run one measurement per sentence, and report them separately. What the rule forbids is the step after, where several measurements are averaged into one number and the number is given the name of the whole.

The practical habit is to write the sentence before the data exists and commit it somewhere you cannot quietly edit. Then the sentence judges the result instead of the result rewriting the sentence.

Receipts: every benchmark in dinostomp carries a one-line question: field, for example "Does the model pick the option MMLU keys as correct?", and the CLI prints it before any result, so the sentence is on screen next to the number. The data-deltas plan opens with its question and five predictions, committed to a git hash before any data was pulled, and the plan file was never edited afterwards.

2Is the metric measuring that, or something nearby that was easier to count?

Word count is not persuasion. Study time is not learning. Lines of code is not productivity. And factual correctness is not honesty: a model can be right and deceptive, or wrong and candid about its uncertainty. If an "honesty eval" only checks whether the facts came out right, it has a name that does not match its metric. Proxies are sometimes all you have; the question is how far the proxy sits from the thing you care about, and whether anyone said so.

The grader is part of the metric. If a person or a model decides what counts as a pass, then two graders have to agree with each other often enough for the score to be about the answers and not about the grader, and the test is blunt: change the grader and see whether the ranking changes. Rubrics that move the ranking when reworded are measuring the rubric.

Receipt: on a published study, two hired annotators labelled a 25% stratified sample and agreed at Cohen's kappa 0.807, every disagreement adjudicated, the result reported with Wilson confidence intervals. Separately, a blind three-rater check (raters blind to condition and to the machine labels, attention checks buried in the form, Fleiss' kappa) cleared one of my two automated classifiers against the human majority and failed the other; the paper reports both. The numbers were not allowed to exist until the graders had been graded.

Receipt: FACTWASH (arXiv:2608.03372) is about rewrites that keep a claim and drop what made it checkable. The rewrite scores as "correct"; the thing that made it true is gone.

3Are the tasks representative of the claim?

If 90% of a "coding" benchmark is beginner Python, a 95% score means "good at beginner Python." The score is not wrong. The label on it is. Read the task list before the leaderboard, every time.

4Is the dataset balanced, or can composition fake the score?

A scam detector tested on 200 scams and 5 normal messages can shout SCAM at everything and score 97%. Class balance is the cheapest thing to check and the most common way a number lies. The same trick hides inside "hard" benchmarks whose hard items are 3% of the rows.

5Did the model see the answers?

If the test set was in the training data, the score measures recall, not ability. This is contamination, and the uncomfortable part is how little it takes. In a pre-registered experiment I planted 500 paragraphs of a test set into a 40-million-token training corpus, 0.2% of the tokens, and trained the same small model with and without them.

Receipt: data-deltas, predictions committed to a git hash before the data was pulled; four of five held. The overlap check in dinostomp flagged 499 of 499 planted paragraphs with zero false hits.

6Outcome or process?

A model can reach the right final answer with reasoning that would fail a first-year student. Final-answer accuracy and reasoning quality are two different measurements. Score both, or say which one you scored. The dangerous case is a benchmark that grades the answer and describes itself as grading the thinking.

7What else could explain the result?

The group with the AI tutor scored higher. They also had human tutors. Now the result has two explanations and the experiment cannot tell them apart. Anything that differs between the groups besides the thing you changed is a confounder, and "we didn't think of it" is not the same as "there isn't one."

8Were the conditions fair?

Model A had a calculator. Model B did not. You tested "A plus calculator against B," not A against B. Same prompt, same tools, same number of attempts, same scoring, or the comparison is not one.

There are two honest comparisons and they answer different questions. Identical setup asks "which model is better under these exact conditions." Each system tuned as well as its owner can manage asks "which product is better." Both are fine. What is not fine is running the second and reporting it as the first, which is how a benchmark table ends up comparing one team's afternoon of prompt work against another team's default settings.

Receipt: Every Verdict We Reported Died to Option Order (paper and harness). The same model, the same questions, the same scorer, and only the order of the answer options changed: verdicts moved up to 986-fold across checkpoints. The condition nobody thought was a condition was the whole result.

9How much does the number wobble on its own?

Two models with the same average can be completely different products.

Seed noise is one source of wobble and the easiest to show. There are others, and they add up. Which items were drawn: a 120-question sample of a benchmark is one draw from it, and a different draw gives a different score without the model changing at all. How many items: a score on 50 items carries an interval wide enough to hide most claimed improvements; the interval should be printed next to the number, and a result whose interval overlaps the baseline's is not a result. Which population: a score on this benchmark is a claim about this benchmark. Whether it carries to the wider set of tasks the benchmark is meant to stand for is a separate claim, and it is usually the one people actually make.

Receipt: the GSM8K finding under question 15 is item sampling, not model randomness: nothing changed but which 120 problems were drawn, and two models moved 11 to 12 points. In data-deltas, the same training data run with three seeds spread 6% to 13% on the benchmark (ledger O-003). A 3% "improvement" from one run at that scale is a coin flip, and the ledger says so (N-001, a prediction of my own that failed inside the noise).

10Compared to what?

45% sounds like failure until you learn that guessing gets 25% and expert humans get 50%.

A score with no baseline is a verdict wearing a measurement's clothes. Ask for random, ask for the simplest possible method, ask for humans. In my forecasting work on retail data, nothing beat the naive seasonal baseline, and that was the finding.

Receipt: The Bar Was Seasonal-Naive, pre-registered holdout, published as the result.

11What do the difficulty bands look like?

An overall score is a weighted average of things that may have nothing in common. Ask for the breakdown. If there is no breakdown, ask why the items were not labelled.

12Does the benchmark cover the claim?

"Coding" is debugging, writing tests, security, architecture, reading an unfamiliar repository, using an API, refactoring, build systems. A benchmark that is entirely "find the syntax error" covers one slice and names the whole. A broad claim needs broad coverage, or a label as specific as the test.

13How are the categories weighted?

Receipt: in MMLU, 78 questions appear verbatim in two subjects and are counted twice in the overall average that every paper reports. Nobody weighted them that way on purpose (hendrycks/test #29, cais/mmlu #35).

14What did it get wrong?

"A scored 80, B scored 75" says less than "A fails the hard items, B fails easy syntax." The pattern of errors is usually the finding. In a drill I worked through recently, a memory-compression eval came back keeping 92% of conclusions and 41% of sources; the useful next step was not another number, it was reading the dropped sources by eye until the pattern showed (sources early in the text were the ones going missing). Numbers tell you that something is happening. Reading the failures tells you what.

Receipt: that drill turned out to be the opening move of Reclaim Evaluation (arXiv:2606.25449), worked example below.

15Which component is actually doing it?

The behaviour you measured could come from the model, the prompt, the tool, the harness around it, the way the context got truncated, or the preprocessing. Change one at a time and watch what moves; researchers call this an ablation. If swapping only the prompt takes a score from 31% to 76%, the failure lived in the prompt, and the next question is whether the fix holds across models or one model happened to like one prompt.

Receipt: dinostomp exists because the instrument, not the model, was the thing moving in my early results: on GSM8K, two 8B models went from 78% to 90% and from 81% to 92% when nothing changed but which 120 problems the seed drew (ledger F-005, $0.06 to reproduce). The sampling was the variable.

The worked example: MMLU, August 2026

MMLU is one of the most-cited benchmarks in the field: 14,042 multiple-choice questions across 57 subjects, reported in model cards since 2021. I ran the questions above against it, mechanically, on a pinned revision of the public data. Exact string equality only, so every count is conservative.

Receipt: cais/mmlu discussion #35 and hendrycks/test #29, with a script that prints every number above from the pinned revision.

A second worked example, from the builder's side: Reclaim

The MMLU example is the questions pointed at someone else's eval. This one is the questions pointed at one I designed, because the checklist is only worth anything if it constrains your own work. Reclaim Evaluation (arXiv:2606.25449) asks whether a language model with a memory can still be corrected after that memory has been compressed.

The claim behind it is simple to say and easy to get wrong: a lossy memory can be worse than no memory at all. A memory that keeps a wrong conclusion but drops the work behind it leads the model to repeat the stale value with confidence; an empty memory leads it to abstain. Whether the model can be corrected turns on one thing, whether the memory kept something it could re-derive the answer from (the source) rather than the answer itself.

The fix that falls out is one line: keep the recomputable source, drop the re-derivable conclusion. At the same budget, that restores correctability where the source is compact enough to keep. The harness, the paired memory conditions and the validators (built to come out false, so a passing run means something) are released with the paper.

Receipt, and a confession: the memory-compression eval sketched under question 14 above, keep the conclusion or keep the source and then read the drops, is the first step of this paper, re-derived from scratch in a study drill a month after I published it. That is what a checklist is for: the questions get you to the same place whether or not you remember the answer.

Two more, from the first review

The first outside read of this page said the fifteen were a good checklist for deciding whether to believe a number and a thin one for deciding whether to act on it, and named what was missing. Both points held up. They are added here as sixteen and seventeen rather than folded into the title, because the title was already published and quietly editing it is what question 1 is about.

16What is the uncertainty, and how far may I generalise?

Every score is a sample. Report the interval, not just the point; say how many items it rests on; and say which population the claim is about. "82% on this benchmark" and "82% on tasks like this" are different sentences, and the second needs evidence the first does not supply.

The habit that goes with this one is reproducibility, because an interval is only meaningful around a run someone else can repeat: the exact model version, decoding settings, prompt, dataset revision, harness commit, number of attempts, and what was done with retries and failures. Each data-deltas run file records its seed, its full configuration, the device it ran on and every evaluation point, and the plan it was scored against is pinned to a commit; the MMLU report pins a dataset revision for the same reason.

Receipt: the annotation study under question 2 reported its agreement with Wilson intervals on a 25% sample, which is what let a small sample stand in for the full set honestly. data-deltas run files, one JSON per run with seed, config, device and the full curve; moons-dont-talk, which writes a receipt binding a training run's verdict to hashes of its data, config, code and environment.

17Does it still hold when the world moves?

A benchmark is a photograph. Users, prompts, languages and adversaries are not in it, and six months from now the model will meet all of them. Before a score is used to decide anything, ask what happens on inputs the benchmark did not sample: another language, a hostile prompt, a formatting change, a population the training data under-represents. A result that has not been tried off the benchmark is a result about the benchmark.

Receipt: When Context Flips (with Dasol Choi) is this question run on model values: the same model, the same question, and a change of framing moves the answer. What a model "believes" on the benchmark is not what it says once the context shifts.

Glossary

The words on this page, in plain language, with the question each one belongs to. Alphabetical.

AblationQ15

Remove or swap one component at a time (the prompt, the tool, the model, the harness) and watch what moves. The way to find out which part is actually producing the behaviour.

AccuracyQ3, 11

The share of items scored correct. Only as meaningful as the items and the scoring rule behind it; an accuracy with no breakdown by difficulty or category is a single number hiding several.

BaselineQ10

The score you would get without the thing being tested: random guessing, the simplest possible method, a previous model, or a human. A score with no baseline attached cannot be called good or bad.

BenchmarkQ3, 12, 13

A fixed, shared dataset plus a scoring rule, used so different models can be compared on the same test. MMLU and GSM8K are benchmarks. A benchmark measures exactly what its items and rule measure, whatever its name says.

Class balanceQ4

How evenly the categories in a dataset are represented. A detector tested on 200 scams and 5 normal messages can score 97% by saying "scam" every time.

ConfounderQ7

Something that differs between the groups being compared besides the thing you changed, and could explain the result on its own. The AI-tutor group that also had human tutors.

Confidence intervalQ16

The range a score would plausibly land in if the same test were run again on a fresh draw of items. Wide on few items, tight on many. Two scores whose intervals overlap have not been shown to differ.

Construct validityQ1, 2

Whether the test measures the thing it claims to measure. A Sudoku score is a valid measure of Sudoku and not of intelligence. The question every other question on this page is a version of.

ContaminationQ5

Test items, or close copies of them, present in the training data. A contaminated score measures memory, not ability. Also called leakage. 0.2% of the training data was enough to cut a benchmark number by 44% on this page.

CoverageQ12

How much of the claimed skill the benchmark's items actually touch. "Coding" covered only by syntax-error items has thin coverage under a broad label.

Difficulty bandQ11

A slice of the benchmark grouped by how hard the items are. Reporting easy, medium and hard separately shows what an overall average hides.

Distribution shiftQ17

When the inputs a model meets in use differ from the ones it was tested on: new users, another language, adversarial prompts, or simply time passing. A benchmark score is a claim about the benchmark's distribution.

Error analysisQ14

Reading the items the model got wrong, by eye, to find the pattern. Numbers say that something is happening; the failures say what.

Eval (evaluation)

A structured test for an AI system: a set of tasks, a scoring rule, enough items to mean something, and a stated claim about what the score measures.

Exact matchQ2

A scoring rule that counts an answer correct only if it equals the reference string exactly. Strict, cheap, and free of judgement; also blind to correct answers phrased differently, which the scorer's owner has to know.

Fixed budgetQ8

Holding a resource constant (tokens of memory, number of attempts, compute) so that only the thing under test varies. Reclaim holds the memory budget fixed and varies only what the memory keeps.

Ground truth (the key)Q2

The reference answer an item is scored against. When the key is wrong, or appears at two option positions, every model is scored against a mistake.

HarnessQ8, 15

The code around the model that builds prompts, calls it, collects outputs and runs the scorer. Part of the instrument, and a common hiding place for the real cause of a result.

Held-out setQ5

Data kept apart from training so that it can be used to test. Only a held-out set the model never saw can measure generalisation.

Inter-rater agreementQ2

How often two graders, human or model, give the same verdict on the same item, usually reported as Cohen's kappa. Low agreement means the score is partly a measurement of who graded it.

Judge-free scoringQ2

Scoring by a fixed rule (exact match, running the tests) rather than by asking another model to grade the answer. Removes the grader's opinions from the measurement; usable only when the correct answer is known.

LLM-as-judgeQ2, 6

Using a language model to grade another model's output. Convenient for open-ended tasks, and a second instrument with its own biases, which now needs its own eval.

Length-matched controlQ8

A comparison condition given the same amount of text as the treatment, so that "more text" cannot be the explanation for an improvement.

PerplexityQ5, 9

A measure of how surprised a language model is by a text; lower means the text was more expected. Standard for pretraining evals. Drops sharply when the model has seen the text before, which is why it exposes contamination.

Pre-registrationQ1, 10

Writing down the question, the method and the predictions before the data is collected, and committing them where they cannot be quietly edited. The result is then judged by the plan instead of the plan being rewritten to fit the result.

Proxy metricQ2

Something easy to count that stands in for the thing you care about. Word count for persuasion, lines of code for productivity. Sometimes unavoidable; always to be declared, with a guess at how far it sits from the real target.

ReproducibilityQ16

Whether someone else, given what you published, gets the same number: model version, decoding settings, prompt, dataset revision, harness commit, attempts and retry policy. A result without these is a report of an event, not a measurement.

RepresentativenessQ3

Whether the items in the test look like the population the claim is about. A coding benchmark that is 90% beginner Python is representative of beginner Python.

ScorerQ2, 8

The rule or program that turns a model output into a score. The most neglected component in most pipelines: it is rarely tested, and when it is wrong every result above it is wrong the same way.

SeedQ9

The random number that decides sampling choices in a run: which items are drawn, how the model samples its words, how weights are initialised. Change nothing but the seed and the score moves; that movement is the noise floor a real effect must clear.

Test setQ5

The items the eval is run on. Ideally a held-out set. The moment a test set is public it starts leaking into training data, which is why fresh items keep having to be written.

Variance (seed band)Q9

How much the score moves across repeated runs that differ only by seed. Two models with the same mean and different variance are different products. Report the spread, not just the centre.

WeightingQ13

How much each category counts in the overall score. A benchmark that is 80% Python debugging is a Python-debugging benchmark whatever else it contains; equal weights are not automatically right either.

The checklist, to keep

1. What exactly is being measured? One sentence.
2. Is the metric valid, or a proxy that was easier to count?
3. Are the tasks representative of the claim?
4. Is the dataset balanced, or can composition fake it?
5. Did the model see the answers?
6. Outcome or process?
7. What else could explain the result?
8. Were the conditions fair?
9. How much does it wobble across seeds?
10. Compared to what?
11. What do the difficulty bands look like?
12. Does the benchmark cover the claim?
13. How are the categories weighted?
14. What did it get wrong?
15. Which component is doing it?
16. What is the uncertainty, and how far may I generalise?
17. Does it still hold when the world moves?