The numbers, and how to reproduce them.
Every figure below comes out of the benchmark directory in the open source repo. Clone it, run the suite, and check our math. We keep two things separate on purpose: hard counts we measured, and scale scenarios we modeled. Both are reproducible. Only one is field telemetry, and neither is yet.
Hard counts and direct measurements.
These are counted or measured directly, not extrapolated.
Gauntlet v2: 100.0 / 100
The full directive and adversarial suite, scored on a fixed rubric. Source: benchmark/gauntlet/.
Context size: 76% smaller
On the compose fixture, naive whole-corpus context is 1,832 tokens. A Perseus compile is 434, and it still answers all four test questions where tuned retrieval drops facts. Source: benchmark/compose/.
1,185 tests passing
The product test suite on the current release. A plain count, run it yourself with pytest.
What happens at scale, computed from the data.
These come from scripts that model larger workloads (tens of thousands of directives, a 500 developer week) using measured per-operation costs. They are reproducible, but they are projections, not production telemetry. We label them that way so you can weigh them honestly.
450× cold to warm
At 50,000 directives a cold render takes 612.6s and a warm one takes 1.36s. The cache absorbs the cost. Source: benchmark/cold-vs-warm.json.
1.0 semantic equivalence
A live judge scored every A/B pair semantically equivalent: same answers from the resolved context as from the raw workspace state. Source: benchmark/ultimate_suite_results.json.
611× cold to warm
22,528 varied directives against the real Perseus repo: cold 619.1s, warm 1.014s. Source: benchmark/real_deltas.json.
Run it yourself.
# clone the repo and run a suite git clone https://github.com/Perseus-Computing-LLC/perseus cd perseus/benchmark # compose: context size and answer coverage python compose/run.py # cold vs warm at scale python run_coldwarm_heavy.py # the gauntlet python gauntlet/gauntlet_orchestrator.py
All data files and scripts live in the repo. If a number here does not match what you get, open an issue and we will fix it.