Perseus Vault scored 0.555 on the MemConflict macro protocol — best of eight memory providers.
The MemConflict benchmark asks an assistant hard questions about facts that changed, false claims planted later in a conversation, and preferences that only apply in specific situations. Perseus Vault answered those questions better than any of the seven other self-hosted memory providers tested — including Honcho, mem0, and Hindsight — while using the fewest model tokens of the field.
Rejected planted false memories at more than double the rate of every other provider — with the fewest false-fact assertions in the field: 5, versus Honcho's 93.
The benchmark plants a false claim after an established fact and asks what the user really said. Perseus Vault scored 0.439 on static conflict and asserted the planted falsehood 5 times in 360 questions. The next-best provider, Honcho, scored 0.181 and asserted the false fact 93 times.
Accuracy at a fifteenth of the token cost.
The benchmark's own headline chart plots answer score against weighted model-token workload — "the better region is upper left." Perseus Vault is the dot in that corner: the top macro score with the lowest token consumption per dialogue turn.
| provider | macro score | weighted tokens* / turn |
|---|---|---|
| Perseus Vault | 0.555 | 739 |
| Honcho | 0.477 | 11,135 |
| mem0 | 0.392 | 3,785 |
| Mnemosyne | 0.142 | 808 |
| Hindsight | 0.281 | 3,779 |
| Supermemory | 0.288 | 2,059 |
| RetainDB server | 0.270 | 6,915 |
| OpenViking | 0.149 | 2,435 |
Field values from the benchmark author's published scores. Perseus Vault row from our replication run (tag v5ftc-perseusvault).
How tokens were measured
Token counts come from the shared vLLM answer-server's own cumulative counters, differenced across the full run window — the same method and the same server every provider used. Uncached input averaged 50.3 tokens per turn (the vault injects only its top-5 ranked memories), and total generated output was 9,783,027 tokens across all 3,750 questions.
Cached input reads 0 because prefix caching was disabled on the serving stack for this wave — a conservative measurement, since cached tokens would only have been cheaper. Embedding-model tokens are not part of this formula for any provider.
Source: token_usage_v5ftc-perseusvault.json · validity: answer-model sidecar valid
Why the low wrong-answer rate matters
The penalty rubric scores wrong answers −1: repeating a stale address or an invented education history is a worse failure than saying "I don't know." Perseus Vault answered 1,996 correct, 358 partial, 1,378 blank, and 18 wrong — the fewest wrong answers in the field, 15× fewer than Honcho's 272. The trade-off is real — 36.7% of questions were left blank rather than guessed — and that abstention is why the false-fact assertion rate stayed so low.
Honest scope: this is a self-run replication of the author's public benchmark — same dataset, models, prompts, and scoring, judged with the author's revised penalty rubric after the author flagged our first run as non-comparable. It is not yet an official inclusion on the author's page. We are asking the benchmark author to add Perseus Vault to the canonical comparison for independent verification; this page will be updated when that happens.
Same conversations, same models, same judge — only the memory provider changed.
Thirty simulated users accumulated 1,579 sessions and 71,060 dialogue turns across a simulated 2022–2026 calendar. Every provider received the same histories and the same 3,750 questions, answered by the same Qwen3.5-4B model with the same gte-modernbert-base embedder, graded by the same Gemma 4 12B judge under a signed penalty rubric. The canonical configuration (temperature 1.0, top_k 5, max_tokens 16,384) was used untouched; the full 30-persona wave completed with zero settle misses.
Every number here is rerunnable.
The replication lives in a public fork of the benchmark repository: scores, manifests, judged checkpoints, serving envelope, and run logs, with sha256 custody hashes recorded. The report's own validator passes clean against the source CSVs.
$ git clone https://github.com/Perseus-Computing-LLC/hermes-memconflict $ cd hermes-memconflict && git checkout 467a71c $ node docs/report/validate-report.mjs $ # artifacts: perseusvault/Scores/ + perseusvault/Results/