LongMemEval-S · the industry benchmark for agent memory

Right memory, 99.8% of the time. 9× less to read.

On 499 of 500 questions, Recallium put the right past session in front of your agent, on a page of 3.8 results where other memory systems hand over 50 to 200. Against the full conversation history, that is about 90% less for your agent to read. It starts from what your team already decided, and skips the pile.

All 500 questions
Abstention included
Reader and judge disclosed
Single pass, no best-of-N

Join the waitlist See how we tested ↓

99.8%

hit@10

right answers in reach from about 90% less reading. More value from every token you pay for.

All 500 questions · 30 Sep 2026

98.6%
recall@10: of all relevant past sessions reach your agent; 95.4%
hit@1: the very first result is already the right one; 3.8
results served: on average, 62% fewer than a standard top ten

3.8 results handed to the reader per question, where Zep hands over 50 and Mem0 200.

Recallium

3.8

Zep

Mem0

200

How it stacks up: more right answers from a fraction of the context.

Recallium hands your agent 3.8 results and gets more answers right than systems that hand over 200. Fewer tokens per call: less to pay for, less for your agent to read.

Vendors ran their tests separately, so every claim below names its protocol, and the table shows what each one disclosed.

3.8results

where Mem0 reads 200

98.4% right from 3.8 results. Mem0 reads 200 to reach 93.4%.

Readers differ: Claude Opus 5.5 here, GPT-5 for Mem0.¹

+19.0pts

over Zep, same rules

Same GPT-4o reader, same judge: 90.2 against the 71.2 in Zep's own paper.

Zep's figure is its peer-reviewed paper result.

+30.0pts

memory beats a bigger prompt

Hand GPT-4o every past session: 60.2. Hand it Recallium: 90.2.

A long context window is not a memory.

+8.2pts

upgrade the model, keep the memory

Swap GPT-4o for Claude Opus 5.5 and the same memory goes from 90.2 to 98.4.

Both readers saw identical results.

Questions answered right, against how much the AI had to read

Most right answers, least to read.

Accurate, but context-heavy

Leaders: accurate and lean

Heavy and lossy

Lean, but lossy

Recallium98.4% · 3.8 results

Zep90.2% · 50 graph items

Mem0¹93.4% · 200 results

No memory system60.2% · entire history, GPT-4o

vs Mem0 at top-50: 98.4 against 90.4, from 3.8 results against 50.¹
vs Zep today: 98.4 against 90.2, from 3.8 results against 50 graph items.

Placed by each system's published QA accuracy and the depth it published (log scale). Readers and judges differ between vendors (see the receipts); the no-memory baseline gives GPT-4o the entire history. Supermemory publishes recall, not accuracy, so it is not placed.

Questions answered right, every system using GPT-4o

Same reader, same judge, 19 points clear.

90.2

71.2

60.2

RecalliumGPT-4o reader and judge

Zep, peer-reviewed paperGPT-4o reader and judge

Full-context baselineno memory system, GPT-4o

Same 500 questions, GPT-4o reader and judge. Recallium's bar uses the official LongMemEval reader and grading prompts. Zep and the full-context baseline are historical paper results, not rerun here.

Why you can trust these numbers

Recallium is the only vendor that states every condition its score depends on. Each mark comes from the vendor's own published material, listed in the receipts.

Disclosure	Recallium	Mem0	Zep	Supermemory
Reader model disclosed	✓	✓	✓	n/a
Judge model disclosed	✓	✓	✓	n/a
Retrieval depth disclosed	✓	✓	✓	◐
Abstention questions: counted or not	✓	✕	✕	✕
Tested configuration stated	✓	✓	✕	✕
Publishes recall and answer accuracy	✓	✕	✕	✕
Conditions stated	6 of 6	4 of 6	3 of 6	0 of 4

✓ stated, ✕ not stated, ◐ partial (Supermemory's “aggregation” is undefined), n/a not applicable to a recall-only figure, and left out of that vendor's total. Vendor rows checked 8 September 2026.

Bring your model. The memory holds.

Recallium supplies the memory; your model writes the answer. Even GPT-4o, reading Recallium's pages, lands within 2.2 points of GPT-4o handed exactly the right sessions. A stronger model turns the same memory into more right answers.

Claude Opus 5.5, agenticRecallium memory

98.4

GPT-4o, official readerRecallium memory

90.2

GPT-4o, handed the right sessionsLongMemEval paper, oracle retrieval

92.4

GPT-4o, best paper retrievalLongMemEval paper

64.0

050100

QA accuracy on 500 questions, GPT-4o judge with the official grading prompts throughout. Both Recallium readers saw identical pages. The paper figures are from Wu et al., 2024, under the same judge.

Per question type

Question type (n)	GPT-4o, official reader	Claude Opus 5.5
single-session-user (70)	97.1	98.6
single-session-assistant (56)	98.2	100.0
single-session-preference (30)	80.0	100.0
knowledge-update (78)	94.9	100.0
multi-session (133)	85.0	98.5
temporal-reasoning (133)	88.0	96.2
of which abstention (30)	76.7	93.3
Overall (500)	90.2	98.4

Abstention: the 30 questions whose right answer is “I don’t know,” counted inside their type as the LongMemEval leaderboard does. Opus 5.5 declined 28 of them. With no memory at all, the same Opus 5.5 reader answers 6.6% of the answerable questions; with Recallium, 98.7%.

Two readers, one memory, one judge: 90.2 and 98.4.

Right on the first result. Complete by the tenth.

The right session usually comes back first, and a relevance gate drops the rest, so your agent reads a short, clean page instead of wading through filler. All 500 questions, abstention included.

95.4%

of questions: the very first result is already a right session

99.8%

have relevant context on the page served, and already in the top five

3.8

results actually served per question on average, not a padded ten

.972

MRR: the right session sits at the top of the page

Retrieval, all 500 questions	Current build	7 Sep build
hit@1	95.4	92.4
hit@5	99.8	97.6
recall@5	98.3	94.0
all-evidence@5	96.0	88.4
hit@10	99.8	98.4
recall@10	98.6	96.3
all-evidence@10	96.8	93.6
precision of the served page	52.6	37.7
results served per question	3.8	6.3
MRR	.972	.946

hit@k: any evidence session retrieved; recall@k: share of evidence sessions retrieved; all-evidence@k: every evidence session retrieved; precision: share of served results that are evidence. One result is one chat session. The current build serves 40% fewer results than 7 September and still lifts every retrieval metric, and the reader reads 59% less: 12.6k tokens per question against 30.7k. “9× less to read” is a different comparison: what the reader was given across all 500 questions, against the full history of each question, on the run that scored 98.4%. That is about 90% less.

Where we're still improving, in the open.

8 misses out of 500 with the Opus 5.5 reader: 3 on retrieval (1 with no evidence served, 2 partial), 3 on the reader and 2 questions answered that should have been declined. We publish every one, because that is where the next gains come from.

Opus 5.5 misses by cause (8)	misses
No evidence served	1
Partial evidence served	2
Reader, with every evidence session on the page	3
Answered a question it should have declined	2

“Reader” means every evidence session was on the page. Partial-evidence misses need several sessions and one was not served.

Weakest types	GPT-4o	Opus 5.5
single-session-preference (30)	80.0	100.0
multi-session (133)	85.0	98.5
temporal-reasoning (133)	88.0	96.2

Preference and multi-session gaps are the reader: both models saw identical pages. Temporal reasoning holds 5 of the 8 Opus 5.5 misses and is the next target; these numbers update when it lands.

The full recipe. Run it yourself.

Every setting behind every number: dataset, metrics, models, dates. One pass per reader, no best-of-N, every per-question output archived.

Protocol

dataset: LongMemEval-S (cleaned), all 500 questions, 30 abstention questions included, no exclusions
corpus: every chat session stored as one Recallium memory, session dates preserved
retrieval: Recallium search, one search per question, serving a page of up to 10 results (3.8 on average)
metrics: session-level hit, recall, all-evidence and precision at k = 1, 5, 10 and MRR, against the official evidence sessions; abstention questions included throughout
readers: Claude Opus 5.5 (agentic harness) · GPT-4o through the official LongMemEval generation script, both on identical pages
blind reading: question ids and memory ids replaced with neutral labels, so no reader can tell from an id which questions are unanswerable
judge: GPT-4o (gpt-4o-2024-08-06), the official LongMemEval per-type grading prompts verbatim, temperature 0, the same judge for every arm
controls: the same Opus 5.5 reader with no memory answers 6.6% of the answerable questions; two further judges agree on 497 to 500 of 500 labels
runs: one reader pass per arm; no best-of-N, no answer ensembling, no prompt-variant voting; per-question outputs archived
date: 30 September 2026

Both lanes reported. Readers and judge named. Depth and dates stated. Abstention included. Official judge, blind reading, one pass.

The receipts

What each vendor has published about its LongMemEval protocol, with sources. Every claim above is checked against this table. A recall number is not comparable to an accuracy number, and the vendor runs were conducted separately, so this is disclosure, not a controlled head-to-head ranking.

System	Headline	Lane	Reader	Judge	Depth	Abstention included	Tested configuration
Recallium	99.8 hit@10, 98.6 recall, 98.4 QA	Recall@10QA accuracy	Claude Opus 5.5 (QA), disclosed; 90.2 with the official GPT-4o reader	GPT-4o, official prompts, disclosed	up to 10, 3.8 on average	Yes, all 30	Recallium search, 30 Sep 2026 build
Mem0	94.4	QA accuracy	GPT-5¹	GPT-5¹	200¹	Not stated	Managed platform, v3 memory pipeline
Zep	90.2	QA accuracy	GPT-5.4	GPT-5.4	20 edges, 10 nodes, 10 episodes, 5 summaries, 5 observations	Not stated	Not stated; the paper build scored 71.2 with GPT-4o
Supermemory	95	Recall@15	n/a, a retrieval metric	n/a	15 + “aggregation”²	Not stated	Source page no longer published²

¹ Mem0’s repository reports 94.4% at top-200 and 94.8% at top-50. The committed result files for those runs name gpt-5 as both answerer and judge, and record 93.4% and 90.4% respectively (results/platform/longmemeval_results.json,longmemeval_top50_results.json); the README figures may come from a later run than the ones stored. Its CLI defaults to GPT-4o, but defaults do not establish what produced a published score. Mem0 retrieves 200 results to reach that stored 93.4%; Recallium reaches 98.4% from 3.8 on average. The readers differ (GPT-5 against Claude Opus 5.5), so read it as a difference in how much context each needs, not as a like-for-like ranking. ² Supermemory’s recall@15 “with aggregation” was published at supermemory.ai/research/longmembench; as of 8 September 2026 that URL redirects to their home page and the figure is no longer published there. The number is retained here as previously reported, and aggregation was never defined. All vendor rows checked 8 September 2026.

Sources

LongMemEval: Wu et al., arxiv.org/abs/2410.10813
Mem0: mem0.ai/research, mem0.ai/blog/ai-memory-benchmarks-in-2026, mem0.ai/blog/mem0-the-token-efficient-memory-algorithm, github.com/mem0ai/memory-benchmarks
Zep paper (71.2%, GPT-4o; full-context baseline): arxiv.org/abs/2501.13956 · Zep 90.2% (GPT-5.4): getzep.com/research
Supermemory (Recall@15 with aggregation): previously at supermemory.ai/research/longmembench, which redirects to their home page as of 8 September 2026. Current research index: supermemory.ai/research
Recallium method and noise analysis: the Memory Benchmark Field Guide. The August 2026 version of this page is kept for the record.

All Recallium figures: LongMemEval-S, full 500 questions, 30 September 2026 build, one reader pass per arm, GPT-4o judge with the official LongMemEval prompts. Vendor figures are self-reported on their own harnesses and quoted with their stated protocols. Corrections: research@recallium.ai.