Result and trade-off
The run produced 1,996 correct, 358 partial, 1,378 blank, and 18 wrong answers. The low wrong-answer count came with a 36.7% abstention rate. Keep that trade-off beside the headline.
| Provider | Macro score | Weighted tokens/turn |
|---|---|---|
| Perseus Vault | 0.555 | 739 |
| Honcho | 0.477 | 11,135 |
| mem0 | 0.392 | 3,785 |
| Hindsight | 0.281 | 3,779 |
Limits
The run used the author's public dataset, models, prompts, scoring, and revised penalty rubric after the author identified a comparability problem. The seven-provider field values come from the author's report; the Perseus Vault row comes from the public replication fork.
Sources
Full Perseus replication report ↗
Benchmark author's canonical report ↗
Replication fork and artifacts status →