Context-Bench pilot: context compilation beats stuffing, at 97% fewer tokens
A 15-question pilot on the public letta-evals filesystem suite, judged by the official strict rubric. The Perseus DAG assembly (perseus#962) outscores full-context stuffing while rendering 2.8% of its tokens, on par with naive RAG arms.
What this measures
Context-Bench is Letta's filesystem-reasoning suite: an agent gets a set of files and strict short-answer questions (rubric.txt, 0 / 0.5 / 1.0). The official target is the Letta Code agent; this pilot adapts the identical dataset and judge to Perseus assembly arms. Canonical results: leaderboard.letta.com.
Arms — full_context: every file stuffed into the answer prompt at the official 8,000-char window. naive_rag_k3/k5: cosine top-k over chunk embeddings. perseus_dag: the auditable context-compilation DAG (perseus#962) — query root → typed evidence gates → selective expansion under a hard token budget, every edge versioned and digest-sealed.
Judge: the official rubric.txt verbatim (SHA-256 9c124fc9…), gpt-5-mini, strict 0/0.5/1.0 parse. Disclosed deviation: the rubric YAML pins temperature 0.0, which the gpt-5-mini API rejects (HTTP 400) — the judge runs at temperature 1.0. Answer model: gpt-5-mini, temperature 1.0.
Tokens: rendered tokens = chars ÷ 4 of what each assembly passes the answer model. Provider-reported usage is captured per call; the full-context arm's real prompt usage averaged ~30k provider tokens per question.
Per-question rubric (n=15)
| question | type | full | rag3 | rag5 | dag |
|---|---|---|---|---|---|
| Among all people who live in the same state as t… | comparison_tiebreak | 0.0 | 0.5 | 0.5 | 0.5 |
| Among people living in the same state as the own… | comparison_tiebreak | 0.0 | 0.0 | 0.0 | 0.5 |
| How many total records (bank accounts, vehicles,… | cross_file_counting | 0.0 | 0.5 | 0.0 | 0.0 |
| Who owns more vehicles: the person with the high… | multi_hop_chain | 0.5 | 0.0 | 0.0 | 0.0 |
| Who owns more vehicles: the person with the most… | multi_entity_comparison | 0.5 | 0.0 | 0.5 | 0.5 |
| Among the 15 people with the highest total bank … | set_intersection | 0.5 | 0.5 | 0.5 | 0.0 |
| Who has a higher total bank account balance: the… | multi_hop_chain | 0.0 | 1.0 | 0.5 | 0.5 |
| What is the total bank balance of the person wit… | aggregation | 0.0 | 0.5 | 0.0 | 0.5 |
| Who has more bank accounts: the person with the … | multi_entity_comparison | 0.5 | 0.0 | 0.5 | 0.0 |
| What is the combined bank balance of the 3 peopl… | aggregation | 0.0 | 0.5 | 0.5 | 0.0 |
| Who has more credit cards: the person with the h… | multi_hop_chain | 0.0 | 0.0 | 0.0 | 0.0 |
| Who has more credit cards: the person with the h… | multi_entity_comparison | 0.5 | 0.5 | 0.0 | 1.0 |
| Who has more internet accounts: the person with … | multi_hop_chain | 0.0 | 0.5 | 0.5 | 0.0 |
| Among people living in the same state as the own… | temporal_reasoning | 0.0 | 0.0 | 0.0 | 0.0 |
| Who has more credit cards: the person with the m… | multi_entity_comparison | 0.0 | 0.5 | 0.0 | 0.5 |
n=15 pilot · gpt-5-mini answer + judge · run 2026-08-15 · 60 answer calls, 60 judge calls, 538,450 provider tokens total (≈ $1 of API spend)
Custody
Every input and the results envelope are digest-sealed and verifiable from the source tree:
pilot sha256 12f842cf270b0bfe3323ea59fb842e2091e6d98fdc6881a84a8961fd7085339a
rubric sha256 9c124fc9cc34841d1bc0c601d13a96246986ca4caaa7730eb3abbd6e8761b385
results digest defee0f5ddcffcd0d027979ff1a2f7a732d8c2faa29bf8cb523e3be19a74df65
Verify with python3 benchmark/context-bench/custody.py in Perseus-Computing-LLC/perseus. Rerun the live pilot with python3 benchmark/context-bench/run.py (requires an OpenAI-compatible key; --dry-run reproduces the full pipeline at zero spend).