Benchmarks

96.3% recall at depth ten on LongMemEval-S.

Results on LongMemEval-S, the industry benchmark for long-running agent memory. Reader, judge, depth and dates are stated for every number.

98.4%
hit@10: an evidence session in the top ten; 96.3%
recall@10: of all evidence sessions served; 92.4%
hit@1: the first result is already right; 93.4%
QA accuracy: Claude Opus 5 reader, GPT-4o judge

How it stacks up

Three vendors publish LongMemEval numbers; each measures something different, and a recall number is not comparable to an accuracy number. The table lists what each has stated about its protocol. The charts distinguish metric types, but the vendor runs were conducted separately and do not establish a controlled head-to-head ranking.

System	Headline	Lane	Reader	Judge	Depth	Abstention included	Tested configuration
Recallium	96.3 recall · 93.4 QA	Recall@10QA accuracy	Claude Opus 5 (QA), disclosed; 82.8 with GPT-4o	GPT-4o, disclosed	10	Yes, all 30	Exact Cloud configuration, run locally
Mem0	94.4	QA accuracy	GPT-5¹	GPT-5¹	200¹	Not stated	Managed platform, v3 memory pipeline
Zep	90.2	QA accuracy	GPT-5.4	GPT-5.4	20 edges · 10 nodes · 10 episodes · 5 summaries · 5 observations	Not stated	Not stated; the paper build scored 71.2 with GPT-4o
Supermemory	95	Recall@15	n/a, a retrieval metric	n/a	15 + “aggregation”²	Not stated	Source page no longer published²

¹ Mem0’s repository reports 94.4% at top-200 and 94.8% at top-50. The committed result files for those runs name gpt-5 as both answerer and judge, and record 93.4% and 90.4% respectively (results/platform/longmemeval_results.json,longmemeval_top50_results.json); the README figures may come from a later run than the ones stored. Its CLI defaults to GPT-4o, but defaults do not establish what produced a published score. That stored 93.4% is the same accuracy Recallium reaches here, but Mem0 retrieves 200 results to get it and Recallium serves 10; the readers differ (GPT-5 against Claude Opus 5), so read it as a difference in how much context each needs, not as a like-for-like ranking. ² Supermemory’s recall@15 “with aggregation” was published at supermemory.ai/research/longmembench; as of 8 September 2026 that URL redirects to their home page and the figure is no longer published there. The number is retained here as previously reported, and aggregation was never defined. All vendor rows checked 8 September 2026.

RecalliumGPT-4o reader · GPT-4o judge

82.8

Zep, peer-reviewed paperGPT-4o reader · GPT-4o judge

71.2

Full-context baselineno memory system, GPT-4o

60.2

050100

Separately reported QA accuracy on the same 500 questions with a GPT-4o reader and judge. Zep and the full-context baseline are historical paper results, not rerun here.

Recallium hit@10Cloud config, local run · 7 Sep

98.4

Recallium recall@10Cloud config, local run · 7 Sep

96.3

Recallium all-evidence@10every evidence session served

93.6

050100

Recallium’s own retrieval at depth ten, run locally with the exact Cloud configuration. No vendor bar is shown: the only comparable retrieval figure was Supermemory’s recall@15, and the page publishing it is no longer available.

Five readers, one memory

Swap only the model that writes the answer; keep the judge. The four August arms saw byte-identical context on all 500 questions. The Opus 5 arm ran on the current build, whose pages score higher on retrieval (all-evidence@10 93.6 against 91.8). That comparison changes both the reader and retrieval build, so it does not isolate the effect of either change.

Claude Opus 5, agenticcurrent build · 7 Sep

93.4

Claude Sonnet, agentic28 Aug build

88.0

GPT-5.6 Sol28 Aug build

87.0

GPT-5.6 Luna28 Aug build

84.8

GPT-4o28 Aug build

82.8

050100

Overall QA accuracy on 500 questions, GPT-4o judge throughout. August arms: hit@10 97.4 · recall@10 94.9 · all-evidence@10 91.8. Opus 5 arm: hit@10 98.4 · recall@10 96.3 · all-evidence@10 93.6. Claude Sonnet re-run on the 31 August build scored 89.2.

Per question type

Question type (n)	GPT-4o	GPT-5.6 Luna	GPT-5.6 Sol	Claude Sonnet	Claude Opus 5
single-session-user (64)	93.8	95.3	96.9	96.9	98.4
single-session-assistant (56)	98.2	94.6	96.4	100.0	100.0
single-session-preference (30)	60.0	50.0	63.3	66.7	90.0
knowledge-update (72)	94.4	93.1	91.7	95.8	97.2
multi-session (121)	72.7	78.5	80.2	79.3	86.0
temporal-reasoning (127)	84.3	84.3	89.0	87.4	92.1
abstention (30)	60.0	86.7	80.0	86.7	100.0
Overall (500)	82.8	84.8	87.0	88.0	93.4

Abstention = the 30 questions whose only correct answer is “I don’t know.” Opus 5 declined correctly on all 30; GPT-4o on 60.0%. On the 468 questions where every evidence session was on the page, Opus 5 scored 97.2%. The Opus 5 column was measured 7 September on the current build, the other four on 28 August.

Five reader configurations, one judge: 82.8 to 93.4.Scores reflect the dated reader and retrieval configurations shown above.

Retrieval you can measure at any depth

A recall number needs a depth attached, and recall without precision hides how much the reader has to wade through. Here is the Cloud configuration tested locally, at every cut. Session-level, all 500 questions, abstention included.

Depth	hit@k	recall@k	all-evidence@k	precision@k	results served
k = 1	92.4	59.5	32.4	92.4	1.0
k = 3	96.6	90.1	81.8	57.4	2.9
k = 5	97.6	94.0	88.4	44.6	4.2
k = 10, what the reader sees	98.4	96.3	93.6	37.7	6.3
k = 10, 28 August build	97.4	94.9	91.8	—	6.1

hit@k: at least one evidence session retrieved · recall@k: fraction of evidence sessions retrieved · all-evidence@k: every evidence session retrieved · precision@k: the fraction of results actually served at depth k that are evidence sessions. Each served result is one chat session. Ranking quality on the current build: MRR .946 · nDCG@10 .933 · MAP .908 (August build: .872 · .878 · .839). At depth one the right session is already there 92.4% of the time; depth ten adds coverage of the questions that need several sessions.

Where we lose points

Thirty-three misses remain with the Opus 5 reader. Twenty of them are retrieval: six pages with no evidence session at all, fourteen with some but not all. Thirteen are the reader’s. Multi-session questions carry most of both.

Opus 5 misses by cause (33)	no evidence served	partial evidence	reader
multi-session (17)	2	9	6
temporal-reasoning (10)	3	5	2
single-session-preference (3)	0	0	3
knowledge-update (2)	0	0	2
single-session-user (1)	1	0	0

A miss counts as “reader” when every evidence session was on the page. The partial-evidence misses are sums and counts across three or four sessions where one was not served. Retrieval improvements shipped since August lifted all-evidence@10 on the 32 window-bearing temporal questions from 59.4 to 75.0, and temporal accuracy from 87.4 (Sonnet) to 92.1 (Opus 5).

Weakest types, five readers	low	high
single-session-preference (30)	50.0	90.0
multi-session (121)	72.7	86.0
temporal-reasoning (127)	84.3	92.1

Preference questions are the most reader-sensitive type: the evidence is retrieved on every one (recall@10 100%), and accuracy runs from 50.0 to 90.0 depending on who reads it. Multi-session questions lose points when one of several evidence sessions falls outside the served page; that is the next retrieval target, and the numbers here will be updated when it lands.

The full recipe

Dataset, metrics, models, and settings used in these runs.

Protocol

dataset: LongMemEval-S, all 500 questions, 30 abstention questions included, no exclusions
environment: Run on a local machine using the exact Recallium Cloud configuration; these are retrieval and answer-quality measurements, not hosted-service latency or availability measurements.
corpus: 23,866 sessions ingested as memories → 86,589 searchable records; session dates preserved
retrieval: Exact Recallium Cloud configuration, serving a page of 10 results per question
metrics: session-level hit, recall, all-evidence and precision at k = 1, 3, 5, 10; MRR, nDCG@10 and MAP over the served page; abstention questions included throughout
readers: GPT-4o · GPT-5.6 Luna · GPT-5.6 Sol · Claude Sonnet · Claude Opus 5 (agentic harness), identical prompts, temperature 0 where the API allows it
judge: GPT-4o, official LongMemEval per-type grading prompts, the same judge for every arm
runs: single full pass per arm; no best-of-N, no answer ensembling, no prompt-variant voting; per-question outputs archived
dates: 28 August 2026 (GPT-4o, Luna, Sol, Sonnet, on the previous build; their retrieval row is shown separately) · 7 September 2026 (Cloud-configuration retrieval, Claude Opus 5)

Both lanes reported. Readers and judge named. Depth and dates stated. Abstention included. Cloud configuration, local run, single pass.

Sources

LongMemEval: Wu et al., arxiv.org/abs/2410.10813
Mem0: mem0.ai/research · mem0.ai/blog/ai-memory-benchmarks-in-2026 · mem0.ai/blog/mem0-the-token-efficient-memory-algorithm · github.com/mem0ai/memory-benchmarks
Zep paper (71.2%, GPT-4o; full-context baseline): arxiv.org/abs/2501.13956 · Zep 90.2% (GPT-5.4): getzep.com/research
Supermemory (Recall@15 with aggregation): previously at supermemory.ai/research/longmembench, which redirects to their home page as of 8 September 2026. Current research index: supermemory.ai/research
Recallium method and noise analysis: the Memory Benchmark Field Guide. The August 2026 version of this page is kept for the record.

All Recallium figures: LongMemEval-S, full 500 questions, Cloud configurations run locally, single pass per arm, GPT-4o judge with official prompts. Vendor figures are self-reported on their own harnesses and quoted with their stated protocols. Corrections: research@recallium.ai.