How we test

The rules

• Outside tests. The headline numbers come from public tests HOBO never trained on, and each number says so. Where HOBO trained on a dataset's training split, we say that too.
• Confident mistakes, counted. Next to accuracy we report how often HOBO is wrong while 90% sure or more. On real tool-call requests that's 2.3%. It's the number that hurts in production.
• Intervals. Scores come with a 95% interval over the test's rows.
• Written down first. Every training run and its pass or fail rules are registered before it starts. Misses are published, and a changed rule only judges runs that come after it.
• Controls. We check that confidence falls when the text is removed or scrambled, so a high score can't come from the answer options alone. With the text removed, HOBO's confidence falls to within 0.1 of an even guess on four of the seven tests we checked, and stays up to 0.16 above it on the others (most on tool calls): see Where HOBO struggles.

Checking the bar holds

A confidence score is only useful if a bar fitted on a small sample still holds on the rest. So we test exactly that: fit a bar on 300 random rows for a 90% target, check it on everything else, and repeat 20 times. A model passes only if the target holds in most splits and the bar still lets it decide a large share on its own. The results are on Fit your bar.

Asking, scored two ways

BFCL, the tool-call test, has no "ask first" answer: it scores an ask as "none fits". We report both. On BFCL's own scoring a bar holds in 20 of 20 splits and HOBO decides 88.8% on its own. Counting an ask as something your system does by itself, and as right only where three independent labellers agreed a detail was missing, it holds in 19 of 20 and decides 89.4%. HOBO had to pass both.

More training, same rules

HOBO is in preview. Any further training run is registered before it starts, read against the same bars, and published here, whether it passes or not.