HOBO docs

Where HOBO struggles

Every limit here has a number behind it. If you hit one we haven't listed, tell us: ask@collapseindex.org.

Measured weak spots

  • Two registered tests missed. This preview missed two conditions we set before training it, and we published it anyway: see Changelog.
    • Reading. It reads SQuAD passages at 90.0%, under the 92.2%we required and the earlier preview's 93.6%. The answers it lost, it gave unsure: none of its reading answers was wrong while 90%+ sure, so a fitted bar hands them back.
    • Decisions it never trained on. On five kinds of decision it never saw in training (scheduling, simple board games, the intent behind a message), it averaged 48.5%, against53.7% for the earlier preview, and was never wrong while 90%+ sure.
  • Claims, and their calibration. On supports-or-refutes questions HOBO ranks the evidence well but on its own leans to "supports". It is served with one calibration toward "refutes", fitted on separate practice data and never on a test: FEVER reads 87.9% with it and70.8% without. It applies only when the options are "supports" and "refutes"; claims asked another way get HOBO's own answer. A bar fitted on 300 rows held its 90% target in18 of 20 splits, so its confident claim answers sit around 90% right, not comfortably above.
  • Pushback, on formats it knows and on ones it doesn't. Add "the user is sure the answer is X" to the text, with X wrong, and HOBO switches a right answer to X and stays confident in 0.1% of cases (0.1% for "an expert reviewer"), against 0.4% for a neutral note. Counting every switch, including the ones it then hands back, 1.7% on claims and 1.1% on reading. On a reading format it never trained on (CosmosQA),6.1% of right answers switch under pushback, against 7.9% for a neutral note, but it is only 57% right there to begin with. Keep opinions about the answer out of what you send it.
  • Layout and menu order move it. We rewrote the rows of three tests by code, keeping the evidence, the options and the right answer. Sent as one JSON object instead of text sections, tool-call requests (scored as a plain pick, without asking first) fell from 86.2% to 84.5% right, with wrong while 90%+ sure unchanged at 3.9%. Numbering the options ("1. ...") barely moves reading (90.1% plain, 90.0% numbered). Reversing the order of a tool menu changed 6.3% of picks; respelling tool or parameter names (camelCase or snake_case) changed 1.1% and 0.6%. Send context as plain text, keep options unnumbered, and fit your bar on requests written the way you'll send them.
  • Text removed, still somewhat sure on claims. Given a request with the evidence blanked out, HOBO's confidence falls to within 0.07 of an even guess on six of seven tests. On supports-or-refutes claims it stays 0.35 above it, because the claim calibration pushes every such answer toward "refutes", evidence or not; none of those answers reaches 90%. A fitted bar catches this; a cut set by hand may not.
  • It rarely asks. 40 asks in 1,933 real tool-call requests, where independent labellers marked 260 as missing a detail. It mostly answers "none fits" instead, which is safe but less helpful. On When2Call, a public test of whether to call a tool, ask, or decline, it scores 25.6%, about chance (25%).
  • Referrals count as refusals. A reply that only points to a professional is a refusal under the definitions HOBO learned. Some benchmarks call those answers. If yours does, fit your bar on your own labels.
  • Banking intents are harder. 85.7% on BANKING77, against 95.4% on CLINC150. Fine-grained intents that share words need a careful bar.

By design

  • It picks; it doesn't write. Every answer is one of the options you gave it.
  • Long documents. HOBO trained on inputs up to about 1,500 tokens. With the evidence buried in 4,000 to 8,000 tokens of other text, it checks claims at 49 to 57%(89.5% on the same claims at normal length) and reads at 85 to 86%(92.5%). Its confidence drops with it: of its 35 answers given at 90%+ on those long inputs, none was wrong, so a fitted bar hands most of them back. Split long documents: see Patterns.
  • No checked examples, no honest bar. Its confidence is a starting point on your data. Your bar comes from your own checked examples.
  • Korean is measured on claims only. On KLUE-NLI, a Korean claim test we didn't write, HOBO reads 83.6% of 3,000 claims (82.2โ€“84.9) and is wrong while 90%+ sure on 1.3% of them, under the 5% limit we set before any Korean claim. Korean inputs get no special adjustment. It is trained on its other Korean decisions too, but those aren't measured on an outside Korean test yet.

Future work

  • Reading back to where it was. This preview gave up some reading for its gains elsewhere. The next HOBO protects it.
  • Claims calibrated in training. The claim calibration is served on top of the model today. The next HOBO learns it, so claims asked any way get it.
  • Asking. HOBO still answers "none fits" where asking would help, and the asking it learned doesn't carry to other formats yet.
  • Long documents. Our first runs training on long inputs didn't improve long reading, and we keep working on it. Until then, split long documents into pieces.