Benchmarks

Meko's LoCoMo results for context quality, and how they compare with other memory providers

Meko's context quality is measured on LoCoMo, a public benchmark for long-term conversational memory. With the benchmark's standard grader, Meko scored 85.4% judge accuracy overall, ahead of every other published result graded the same way. The session that holds the answer was in Meko's top 50 results for 97.9% of questions.

About LoCoMo

LoCoMo has ten long conversations between two people and 1,986 questions about them. Like other published results, Meko's results score the 1,540 questions in four types and leave out the 446 adversarial questions:

  • Single-hop (841 questions): the answer is in one session.
  • Multi-hop (282 questions): the answer combines facts from several sessions.
  • Open-domain (96 questions): the answer needs world knowledge as well as the conversation.
  • Temporal (321 questions): the answer depends on when something happened.

Each run loads the ten conversations into a datapack and waits for Meko to finish extracting memories and indexing. An agent then answers each question from what context_search returns, which searches the datapack's memories and recent conversations together.

Each answer gets two checks:

  • Session recall (R@k): whether the session that holds the answer is among the top k results Meko returns.
  • Judge accuracy (J): whether an LLM judge marks the answer correct against the reference answer. The score is the share of questions marked correct.

Session recall

Results returned (k) Session recall, all question types
10 94.3%
25 96.4%
50 97.9%

Session recall measures retrieval only, so no judge is involved.

Judge accuracy

Different judges accept different answers, so judge accuracy scores compare directly only when they come from the same judge. The results below are grouped by judge.

Standard grader

These results use the LLM judge prompt from the Mem0 2025 paper (its Appendix A). Meko run 2 was ahead of every other provider in every question type.

Provider or run Single-hop Multi-hop Open-domain Temporal Overall J Answer model
Meko, run 2 91.8 78.7 69.8 79.1 85.4 gpt-5.6, two-step prompt
Meko, run 3 88.7 78.0 71.9 74.5 82.7 gpt-5.3, two-step prompt
Lyzr Cognis 85.9 72.0 51.0 77.3 79.4 "GPT-4", as published
Mem0, 2025 paper 72.9 67.1 51.2 55.5 66.9 gpt-4o-mini

Overall J is the per-type scores weighted by question count. For the Meko runs, the judge ran on gpt-4.1. Lyzr publishes no overall score, so its overall is calculated from its per-type scores.

The Mem0 2025 paper labels its question types differently from LoCoMo's categories, so this table places its scores by question count. That placement reproduces the paper's own overall of 66.88. Under the paper's labels, its Table 1 shows 67.1 single-hop, 51.2 multi-hop, 72.9 open-domain and 55.5 temporal.

LoCoMo also reports two word-overlap scores, F1 and BLEU-1 (B1). They count the words an answer shares with the reference answer, so a correct answer in different words scores low.

Provider or run Single-hop F1 / B1 Multi-hop F1 / B1 Open-domain F1 / B1 Temporal F1 / B1
Meko, run 2 64.6 / 57.6 47.1 / 37.8 41.4 / 36.5 62.9 / 57.2
Meko, run 3 63.5 / 57.5 47.4 / 36.8 33.0 / 28.2 56.4 / 50.3
Lyzr Cognis 54.8 / 45.1 48.7 / 35.8 31.5 / 23.2 62.7 / 59.0
Mem0, 2025 paper 47.7 / 38.7 38.7 / 27.1 28.6 / 21.6 48.9 / 40.5

Other graders

Each of these results uses its own judge. The Meko baseline run used the judge in Mem0's evaluation code, which is more lenient than the standard grader.

Provider or run Single-hop Multi-hop Open-domain Temporal Overall J Answer model and judge
Zep 96.4 94.0 79.2 95.6 94.7 gpt-5.4, Zep's own chain-of-thought judge and question counts
Meko, baseline run 94.6 92.6 79.2 91.9 92.7 gpt-4.1-mini with Mem0's reasoning prompt and top 25 results, Mem0's evaluation-code judge
Mem0 Platform v3 (May 2026) 94.6 95.4 82.3 92.5 92.5 Mem0's own 2026 judge. Weighting its per-type scores by question count gives 93.5, not the published 92.5
Supermemory - - - - 77.1 Judge and per-type scores not published

Redis has published no LoCoMo result.