Research
Papers, case studies and essays. Each one comes with its code and its findings, including the ones that didn't hold.
Selected Publications
★ If you read one, read this one.
- Reference, Retrieve, Resolve: How Small a Language Model Can Read a Reference in Context
- Coherent Values, or the Frame That Asked? From Preference Transitivity to Identifiability
- Explicit, Not Longer: What Makes Epistemic Stance Survive Memory Compression
- FACTWASH: Catching AI Rewrites That Wash Hearsay into Fact
- Every Signal We Found Was an Artifact: A Calibrated Control Battery for Secret-Loyalty Auditing
- Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak
- They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It
- Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Facts
- Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One
- When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models
Case Studies
- How Small a Model Can Read Your Documents - Deployment Study
the decision memo version of a pre-registered eval: ten models from 1B to 235B answering 400 questions from a reference document supplied in the ...
prompt, then the same 400 with the document taken away. one model clears 90% and it is the only one you cannot run on your own hardware, tune, or pin to a version. every model that works loses a quarter to a third of its accuracy without the document, a larger effect than the gap between any two of them, so the buying decision is retrieval quality rather than model choice. priced against a frontier model at 100,000 lookups a day: even the cheapest frontier tier is 21x more. - Two Tenths of a Percent - Pretraining Experiment
a pre-registered experiment: what dinostomp-style filters do to a small pretraining run, measured against seed noise.
four training sets cut from 160,000 FineWeb documents at one 40M-token budget (raw, deduplicated, dedup plus hygiene, dedup plus 500 planted WikiText-103 test paragraphs), twelve 30M-parameter GPT runs, three seeds each, five predictions committed to a git hash before the data was pulled. four held, one failed and is published. planting 0.2% of the tokens cut the benchmark perplexity 44% while the training loss and the in-distribution held-out sat flat; dinostomp flagged 499 of 499 planted paragraphs with zero false hits; seed noise on the out-of-distribution eval was 6 to 13%, wider than dedup or hygiene moved it. dedup did nothing because FineWeb already did it, and the write-up says so. - The Bar Was Seasonal-Naive - Data Analysis
an audit-first sales analysis and demand forecast of Online Retail II (UCI; 1,067,371 invoice lines of a UK online retailer, 2009-2011).
the audit finds a nine-day byte-identical overlap between the file's two sheets, admin fees posing as products, and warehouse write-offs inside the sales table; the forecast's rules were registered before the data was opened, and nothing beat seasonal-naive (MAE £52,948, 20.0% MAPE on a 13-week Christmas-ramp holdout). published as the finding, not a failure. - Three Rulers, One Dataset - Data Analysis
an audit of MovieLens 25M, one of the most-used public research datasets in the world: 25,000,095 ratings across 25 years.
every structural promise its documentation makes holds exactly, and the timeline underneath is three rating instruments stitched together at seams the documentation never mentions. the oldest instrument flattered bad movies by nearly a full star. the data is immaculate; the ruler moved. - July Orders Review - Data Analysis
an end-to-end analyst engagement on a deliberately dirty e-commerce dataset:
a paste-over that both duplicated one order and destroyed another, a region spelling that silently split every regional rollup, and a sales channel that returned 70% of the revenue it booked. the write-up separates what the data shows from what it can prove, and maps every open question to the record that would settle it. - The Join Is Dirtier Than the Data - Data Analysis
an audit-first analysis of the Animal Crossing: New Horizons catalog (30 tables, 18,962 rows).
the tables are close to immaculate; the string keys between them are not: one capitalization mismatch silently drops three villagers from any song join and manufactures a false finding, three placeholder rows inflate the song catalog, a sentinel string poisons the price column's type in 22 of 25 tables, and a perfect personality-gender partition makes every gender claim a design claim. includes one false alarm by the analyst, corrected and kept on the record.
Essays
- Fifteen Questions Before I Trust a Number
a plain-language guide to evaluation fundamentals: the fifteen questions to ask of any benchmark score before believing it.
what exactly is measured, is the metric a proxy, did the model see the answers, compared to what, how much does it wobble across seeds, which component is doing it. each question carries a receipt from published work (the planted-contamination experiment, the option-order paper, the filed MMLU defect report), and the page ends with two worked examples, MMLU from the auditor's side and Reclaim Evaluation from the builder's, and a plain-language glossary. - Evals Are a Supply Chain Problem
a program management essay on AI evaluation pipelines, written in the dialect of an 11-year consumer-goods supply chain:
shards are pallets sized by affordable damage, idempotency is one BOL slot per pallet, dead shards are red-tagged pallets in the hold cage, and a dashboard that hides its shortages is a distributor scorecard with a secret baseline. pin the definitions before you do anything.