Benchmarks that you can rerun

Agent-memory vendors publish numbers; ours come with the script, the dataset, the named model, and a sha256 signature over the result set. Everything below is generated from committed reports in the open repo. If you cannot reproduce a number, it comes down.

Feature matrix

Architecture facts, not adjectives. Competitor rows come from their public repos (Zep / Graphiti · Mem0 · Cognee · Letta); anything we could not verify is "n/a".

	Perseus Vault	Zep / Graphiti	Mem0	Cognee	Letta
Temporal model	Full SQL:2011 bi-temporal (valid + transaction time, point-in-time recall)	Bi-temporal edges (Graphiti)	Timestamps	Timestamps	Timestamps
Storage / deployment	Single local binary, SQLite, offline	Server + graph DB	Server or SaaS, vector DB	Server, graph + vector DBs	Server, Postgres
Encryption at rest	AES-256-GCM built in	n/a	n/a	n/a	n/a
Audit trail	Hash-chained journal (keyed MAC under encryption; external review pending)	n/a	n/a	n/a	n/a
Retrieval modes	FTS5 keyword + dense vector + hybrid RRF, decay + trust weighted	Graph + semantic	Vector	Graph + vector	Vector + recall memory
License	MIT	Apache-2.0 (Graphiti)	Apache-2.0	Apache-2.0	Apache-2.0

Retrieval recall (LongMemEval)

Session-level recall against the 500-instance LongMemEval split (23,867 sessions ingested into the real binary, offline, judge-free; default hybrid retrieval). perseus-vault.exe.

84.6%

recall@1

95.2%

recall@3

97.4%

recall@5

99.2%

recall@10

0.903

MRR

source: benchmark/longmemeval/report.json sig 9babb85f54f9

End-to-end QA vs Zep

Zep publishes 63.8% on LongMemEval with GPT-4o (their paper). Ours below: identical split, pinned answerer and judge named in the report, LongMemEval's official per-type judge prompts. The headline is the mean of 3 independent signed runs (73.6% / 75.0% / 72.8%); the worst run scores +9.0 points vs Zep's published number. Where conditions differ from a competitor's published run, the comparison is flagged, not blended. Per-type table is from the primary run's signed report.

73.8%

mean of 3 independent full runs (range 72.8–75.0%, gpt-4o-2024-08-06)

79.0%

with LongMemEval's official CoT answer prompt (answer_prompt: official-cot) — mean of 3 independent signed full runs (range 78.4–80.0%)

The benchmark ships two official answer prompts (plain and step-by-step CoT); both distributions above are 100% official methodology and differ only in that flag — each number carries its answer_prompt, recorded in the signed report. Zep's publication does not state which variant they used, so the comparison is flagged, not blended.

source: benchmark/longmemeval/qa_report_cot.json sig 20327b31b594

source: benchmark/longmemeval/qa_report_cot_seed2.json sig 7c8ce1b406c0

source: benchmark/longmemeval/qa_report_cot_seed3.json sig eb848e786677

question type	correct	accuracy
knowledge-update	60/78	76.9%
multi-session	84/133	63.2%
single-session-assistant	56/56	100.0%
single-session-preference	9/30	30.0%
single-session-user	67/70	95.7%
temporal-reasoning	92/133	69.2%

source: benchmark/longmemeval/qa_report.json sig 929623670d8b

source: benchmark/longmemeval/qa_report_seed2.json sig 4e816cf0a904

source: benchmark/longmemeval/qa_report_seed3.json sig c712d867a958

Same-box recall, all systems fully local

Every system stood up and run live on one H100 against the same local Ollama (qwen2.5:14b-instruct + nomic-embed-text): identical fact set, identical queries, identical substring judge. No published or cloud numbers are substituted for a competitor. Cells show recall accuracy and p50 latency.

system	recall	p50 latency	stack
Perseus Vault (hybrid)	1.00	35.6 ms	single local binary, in-process
Letta (archival / pgvector)	1.00	135.5 ms	server + Postgres/pgvector
Mem0 (vector)	0.60	37.9 ms	Python + Qdrant
Zep (Graphiti temporal KG)	0.20	49.7 ms	server + Neo4j

Letta matches Perseus Vault on recall for this corpus; the difference is operational, an in-process binary answering in ~35 ms versus a server plus Postgres at ~135 ms p50. Zep's self-hosted Community Edition server is deprecated and its zep_python memory API is now Zep Cloud only, so this measures Zep's open-source engine, Graphiti (a temporal knowledge graph on Neo4j), with entity extraction and embeddings both on the same local Ollama. Building a knowledge graph needs an LLM to do structured extraction, and a local model is lossy at it (5 entities and 2 edges from 6 facts here), so 0.20 reflects local-extraction quality rather than Zep Cloud, which uses frontier models. The point of this table is not a scoreboard; it is what each architecture does when the whole stack has to run locally.

source: benchmark/lambda/results/competitors.json · competitors_bench.py

Scale and latency

The real binary over MCP stdio, seeded corpus, 100 queries per metric. Hardware named in the report: Windows 11, 16 cores. Bi-temporal point lookups stay flat from 10K to 100K entities; that column is the differentiator.

entities	write sustained	fts5 p50/p99	dense p50/p99	hybrid p50/p99	as_of p99	cold start
10,000	479/s	3.14 / 9.62 ms	11.96 / 14.29 ms	19.03 / 24.56 ms	0.53 ms	56.6 ms
100,000	40/s	15.67 / 23.23 ms	25.43 / 32.78 ms	79.73 / 92.86 ms	0.46 ms	58.2 ms

source: benchmark/scale/report.json sig c688a8490cc1

Temporal correctness

Bi-temporal reconstruction checked scenario by scenario: what was believed at T (transaction time), what was true at T (valid time), and the full cell. Deterministic, offline.

100.0%

gauntlet: 13/13 checks

10/10

valid_at

1/1

as_of

2/2

bitemporal

source: benchmark/temporal/gauntlet_report.json sig 2c1e38ab18aa

100.0%

temporal suite: 20/20

source: benchmark/temporal/report.json sig 009c4c3e25cb

Correctness and determinism at scale (BEAM)

Named for BEAM (Beyond a Million Tokens), which tests at 128K / 500K / 1M / 10M tokens to defeat the "dump everything in context" cheat. Our claim is orthogonal to a context-window score: FTS5 + deterministic bi-temporal retrieval must not degrade as the corpus grows. BEAM embeds the CI-verified bi-temporal gauntlet (the same 13 checks) inside a filler corpus sized to each token tier and asserts, at every tier, that correctness holds (13/13), that two independent runs produce an identical signature (deterministic=true), and that point lookups stay flat. Fully offline, lean FTS5 + bi-temporal binary (no embeddings), on Linux / Unraid.

100.0%

gauntlet: 13/13 at every tier (128K–10M)

deterministic=true

identical signature across two runs, every tier

23.07 ms

as_of p50 at 10M (flat from 22.08 ms at 128K)

10M

153,846 entities, ~10.1M tokens, 332.43s populate

token tier	filler entities	approx tokens	populate	gauntlet	deterministic	as_of p50/p95	valid_at p50/p95
128K	1,969	129,461	2.87 s	13/13	yes	22.08 / 24.36 ms	22.75 / 25.61 ms
500K	7,692	505,749	10.21 s	13/13	yes	21.26 / 24.45 ms	17.7 / 24.11 ms
1M	15,384	1,011,498	27.58 s	13/13	yes	23.17 / 88.83 ms	22.09 / 23.95 ms
10M	153,846	10,115,374	332.43 s	13/13	yes	23.07 / 24.61 ms	22.62 / 25.56 ms

Point-lookup p50 is flat (22.08 → 21.26 → 23.17 → 23.07 ms as the corpus grows 128K–10M), because lookups are (category,key)-indexed. p95 is flat too except a single 88.83 ms sample at the 1M tier, shown as measured; every other tier stays under 26 ms. 40 samples per axis per tier.

source: benchmark/beam/report.json sig 2c1e38ab18aa · methodology

Reproduce every number

Every report above is generated by a script in the repo and signed with a sha256 over its result set. From a clean clone:

git clone https://github.com/Perseus-Computing-LLC/perseus-vault.git
cd perseus-vault
cargo build --release
python benchmark/recall/run.py        # recall quality (offline)
python benchmark/longmemeval/run.py   # LongMemEval retrieval (offline)
python benchmark/temporal/gauntlet.py # bi-temporal gauntlet (offline)
python benchmark/scale/run.py         # scale + latency (offline)
python benchmark/beam/run.py --tiers 128K 500K 1M 10M --out /tmp/beam.json  # BEAM: correctness + determinism at scale (offline)
python benchmark/longmemeval/qa.py    # end-to-end QA (needs an OpenAI key; prints cost first)

This page was generated by scripts/gen_benchmark_page.py at commit eb8bc17. If a number here cannot be traced to a committed signed report, that is a bug; please file it.