Skip to content

How AI memory benchmarks work

A memory benchmark gives a memory system a long conversation, asks questions about it, and grades the answers. The score means something only next to the setup that produced it.

The short version

A memory system is the part of an AI agent that keeps what was said in earlier sessions and hands the relevant pieces back later, when they matter. To test one, you need a long conversation, questions about it, and answers you already know.

The memory system takes the conversation in once. Then, one question at a time, it hands back what it thinks is relevant, a language model writes an answer from that, and a second model, the judge, checks the answer against the one you know. The score is the share of answers the judge marks correct.

Once, before any question

  1. DatasetA long conversation. Its questions and reference answers are set aside.
  2. Memory system stores itIt takes the conversation in, in its own way.

Then, for each question

  1. QuestionAbout something said earlier in the conversation.
  2. Memory system retrievesIt hands back what it finds relevant.
  3. AnswererA fixed language model writes an answer from that.
  4. JudgeA second model checks it against the reference answer.
  5. ScoreThe share of answers the judge marks correct.

Outlined in ink: the memory system, the only part that changes from one run to the next.

The test pipeline. The memory system takes in the conversation once. Each question then goes through the same steps: the memory system retrieves, a fixed answerer writes an answer, and a judge grades it against the reference.

Three things move that score besides the memory system: the model that writes the answer, the judge that grades it, and how much text the memory system may hand over. Change any of them and the score changes, even when the memory system is the same. That is why a score without its setup tells you little, and why two scores from different setups cannot be lined up.

What a LoCoMo score measures

LoCoMo is a public dataset of very long conversations between two people, from the 2024 paper Evaluating Very Long-Term Conversational Memory of LLM Agents by Maharana and colleagues. The conversations were generated by language model agents and checked by people. The paper describes 50 of them, each about 300 turns and 9,000 tokens long on average, spread over as many as 35 sessions. The data file the authors published holds 10 conversations and 1,986 questions.

Every question is one of five kinds, and each comes marked with the turns of the conversation that hold its answer:

  • Single-hop: the answer is in one session.
  • Multi-hop: the answer needs information from several sessions.
  • Temporal: the answer depends on when something happened.
  • Open-domain: the answer combines what a speaker said with general knowledge.
  • Adversarial: the question is built to trick the system into a wrong answer, and the right response is to say the conversation does not answer it.

So a LoCoMo score is the share of those questions answered correctly when one particular memory system supplies the context, one particular model writes the answers, and one particular judge grades them. It measures the whole chain, not the memory system alone.

Two details change what the number means. The paper graded answers by word overlap with the reference answer, an F1 score, while many later setups ask a language model to judge them instead, and the two kinds of score are not interchangeable. And a setup can leave the adversarial questions out, which scores 1,540 questions instead of 1,986. Before you read a LoCoMo score, find out how it was graded and which questions it counts.

LoCoMo vs LongMemEval

LongMemEval is a second public dataset for the same question, built differently. It comes from the paper LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory by Wu and colleagues, published at ICLR 2025. It has 500 questions, and each one comes with its own chat history between a user and an assistant. It tests five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention, which means declining to answer when the history does not hold the answer.

Its histories come in three sizes. The oracle version holds only the sessions that contain the answer. The S version adds sessions that do not, for about 50 sessions and roughly 115,000 tokens per question. The M version has 500 sessions, around 1.5 million tokens. Every session carries a date, and so does the question. The authors later published a cleaned version that removes noisy sessions from the histories, so a LongMemEval score also depends on which release it used.

LoCoMo and LongMemEval, side by side.
PropertyLoCoMoLongMemEval
PaperMaharana and colleagues, 2024Wu and colleagues, ICLR 2025
HistoryLong conversations between two peopleA user and an assistant, one history per question
SizeAbout 300 turns and 9,000 tokens per conversation, over up to 35 sessionsAbout 115,000 tokens per question in S, around 1.5 million in M
Questions1,986 in the published file, over 10 conversations500
What it testsSingle-hop, multi-hop, temporal, open-domain, adversarialInformation extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention
Where the answer is markedThe turnThe session, with the answering turns flagged inside it
Graded in the paper byWord overlap (F1)A language model judge

The difference that matters most for testing is size. A LoCoMo conversation fits in the context window of nearly any current model, so on LoCoMo a memory system competes with simply pasting the whole conversation in. LongMemEval M is larger than most models can read at once, which is the situation memory is for. The LongMemEval paper reports that commercial chat assistants and long-context models show a 30% drop in accuracy when they have to remember information across sustained interactions.

Neither dataset covers all of what an agent has to remember. Both are built from chat, and neither tests what a coding agent carries from one session to the next, such as a decision and the reason behind it. Facts that change over time get a share of their own in LongMemEval: 78 of its 500 questions are knowledge updates.

The answerer and the judge

A memory system does not answer questions. It hands back text. Two more models turn that text into a score, and both are part of what gets measured.

The answerer

The answerer is a language model that reads what the memory system returned and writes the answer. For a comparison to mean anything, every memory system gets the same answerer: the same model, the same settings and the same prompt.

A very strong answerer is not better here. It can reason its way to a right answer from thin, scattered context, which hides the difference between a memory system that found the evidence and one that nearly did. A modest answerer, held fixed, lets those differences show.

The prompt matters as much as the model. A prompt that asks for step-by-step reasoning gets different answers from one that asks for a single sentence. A prompt written with short facts in mind can also penalise a memory system that returns long passages, for the prompt’s sake rather than the memory system’s.

The judge

The judge is a second language model that compares each answer with the reference answer and marks it correct or not, or gives it a grade. It is a measuring instrument, and like any instrument it has to be checked. Four questions decide whether a judge can be trusted:

  • How strict is its rubric? A rubric that counts an answer when it touches on the same topic as the reference gives credit for being nearby. A strict one asks for the fact itself.
  • Was it checked against people? Before a judge is trusted, its grades on a sample of answers should be compared with grades from people. Judges differ, and some small models grade too leniently.
  • Is it from a different model family than the answerer? A model grading writing from its own family is a known source of bias.
  • Is it pinned? The judge’s model version, prompt and settings should be recorded and held fixed, so the run can be repeated.

Invented example.

Question
When did Dana start working at the bakery?
Reference answer
March 2023
Strict rubric
The answer has to state the fact in the reference.
Generous rubric
An answer on the same topic as the reference counts.
Four answers to the invented question, each graded under the strict and the generous rubric.
AnswerStrictGenerous
“She started in March 2023.”CorrectCorrect
“She started in the spring of 2023.”WrongCorrect
“She talked about a new job at a bakery.”WrongCorrect
“I don’t have that information.”WrongWrong
Score1 of 43 of 4
The same four answers, graded by two rubrics. The generous one scores three times higher, and nothing about the memory system changed.

A judge also has its own error, so two scores a point apart may be the same score once that error is counted. A small set of answers graded by people can measure how far the judge is off and correct the whole run for it, a method called prediction-powered inference. Resampling the questions, a bootstrap, puts a confidence interval on scores that already exist, at no extra model cost.

The full-context baseline

The simplest setup has no memory system at all: put the whole conversation into the answerer’s prompt and ask the question. That is the full-context baseline. It runs with the same answerer, the same judge and the same questions as the memory system under test, which is what makes it an anchor.

Read against it, a memory system’s score says something. Below the baseline, the memory system left out something the answerer could have used. Above it, the memory system helped, usually by leaving out what distracted the answerer. Without the baseline beside it, the same score could mean either.

Illustration. The values are invented, not measured.

On a short history

No memory, the answerer alone
Memory system
Full context, the whole history in the prompt

Below the anchor: the memory system left out something the answerer could have used.

On a very long history

No memory, the answerer alone
Memory system
Full context, the whole history in the prompt

Above the anchor: the memory system helped, by leaving out what distracted the answerer.

The memory system’s bar is the same length in both panels, and it means two different things. Only the full-context baseline, run with the same answerer and judge, tells you which. The outlined bars are the two baselines; the filled bar is the memory system.

A second anchor sits at the bottom: the same answerer with no memory at all. It shows how much the answerer gets right from general knowledge or by guessing. On questions whose right answer is that the conversation never said, an answerer with nothing to go on can score by declining, which is one reason abstention is reported on its own.

Full context is an anchor, not always a ceiling. The 30% drop in the LongMemEval paper includes long-context models, given the history to read. Once a history is bigger than the model’s context window, as LongMemEval M is for most models, there is no full-context run to make. And it is the most expensive way to answer, since every question pays for the whole history.

The metrics

A careful benchmark reports several numbers and labels each one as measured or judged, because they fail in different ways.

Answer accuracy, judged

The share of answers the judge marks correct, and the number most often published. It belongs per kind of question, not only as one average: a system can be strong on single-hop questions and weak on temporal ones, and an average across different abilities hides which.

Retrieval, measured

Both datasets mark where each answer sits, so you can check whether the memory system handed the answer back at all, with no judge involved. Recall at k asks whether the evidence is among the first k results; when a question has several pieces of evidence, it is the fraction found. Reciprocal rank is one divided by the position of the first piece of evidence, so first place scores 1. Neither moves when the judge changes.

Invented example.

Question
What did Sam name the dog?
Evidence
turn 214, the turn the dataset marks as holding the answer

What the memory system returned, in order

  1. 1turn 88“We should get a dog someday.”
  2. 2turn 301“The dog park was packed this morning.”
  3. 3turn 214“We named him Biscuit.”Evidence
  4. 4turn 15“My sister has two cats.”
  5. 5turn 140“Biscuits are in the cupboard.”
recall@1
0The evidence is not the first result.
recall@3
1The evidence is in the first three.
recall@5
1The evidence is in the first five.
reciprocal rank
1/3The evidence came back third.
Retrieval scores need no judge. They only ask where the marked evidence came back in the list, so they stay put when the answerer or the judge changes.

The two datasets mark evidence at different levels of detail. LoCoMo’s evidence is a list of turns; LongMemEval’s is a list of sessions, with the answering turns flagged inside them. A retrieval score depends on the level it is counted at, so a retrieval score on one is not a retrieval score on the other. And LongMemEval’s oracle version is a weak test of retrieval: it leaves out every distracting session, so there is nothing wrong to pick, and recall there only says how much of the evidence came back, not whether it could be found among noise.

Abstention

When a question has no answer in the history, did the system decline? And when it does have one, did the system decline anyway? Both are counted, because a system that refuses often looks careful on the first and fails the second.

Error bars

A score from a few hundred questions has a margin. A published difference smaller than that margin, or one reported from a single run with no interval, is not yet a difference.

Cost, tokens and the context budget

Every question is paid for twice: once for the answerer to read the retrieved text and write an answer, and once for the judge to grade it. The more text a memory system hands back, the more each answerer call costs.

That amount also drives the score. An answerer that sees more text has more chances to find the answer, so a memory system that returns a lot can look better for its volume rather than its memory. Asking every system for its top 10 results does not fix this: ten short facts and ten whole sessions are both a top 10.

Illustration. The values are invented, not measured.

System A, top 10: ten short facts
System B, top 10: ten whole sessions

Dashed line: a budget both systems share. The outlined part of a bar is the text over it.

Both systems returned ten results. System B handed the answerer many times more text, which costs more on every question and can raise its score for volume alone. Held to a shared budget, System B has to choose what to send.

A 2026 preprint, Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration, shows how far this goes, on memory for scientific papers. The system that led its uncapped run was returning about 2.55 million characters per question. Compared at a fixed budget of about 30,000 characters, that lead disappeared. The authors also call budget control “necessary but not sufficient”: on their second dataset, the gap between systems did not vanish under the budget.

So a fair setup records, for every question, how much text the memory system returned, in characters as well as tokens, since the same text counts as a different number of tokens under different models. It either holds every system to the same budget or prints the budget next to the score.

Two costs never show in an accuracy score. Some memory systems call a language model for every message they store, to pull facts out of it, so they pay at write time, before any question is asked. And retrieval takes time: a system that answers well but slowly may not suit an agent that has to respond within a second. Both belong next to the score.

Why published scores can’t be compared

Memory systems are often announced with a LoCoMo or LongMemEval score, and it is tempting to line those numbers up. Two scores are comparable only if everything except the memory system was the same, and published setups often differ in:

  • the answerer model and its prompt;
  • the judge model, and how generous its rubric is;
  • which questions are counted, such as whether the adversarial ones are left out;
  • which version of the dataset was used, such as LongMemEval’s oracle, S or M, original or cleaned;
  • how much text the memory system could hand the answerer;
  • whether the score is one run or several, and whether it carries an interval;
  • who ran it. A team testing its own system makes every one of these choices.

Illustration. The bar lengths are invented, not measured.

Two setups for testing the same memory system on LoCoMo, and how they differ.
Same memory system, on LoCoMoSetup ASetup B
AnswererA modest model, one-sentence answersA larger model, step-by-step reasoning
Judge rubricStrict: the answer states the factGenerous: the same topic counts
Questions countedAll 1,9861,540, adversarial ones left out
Text per questionCapped at a shared budgetNo cap
RunsSeveral, with an intervalOne
Published score, setup A
Published score, setup B
Every row is a choice that can move a score, and most of setup B’s make a high number easier to get. The memory system is the same in both, so the gap between the two bars is a gap between setups.

None of this means a published number is wrong. It means the number belongs to its setup. A gap between two systems’ scores, taken from two different setups, may be a gap between the setups.

What to ask before comparing two scores

  1. Which dataset, which version, and which questions were counted?
  2. Which model wrote the answers, and with which prompt?
  3. Which model judged them, with which rubric, and was it checked against people?
  4. What did the full-context baseline and the no-memory floor score in the same setup?
  5. How much text did the memory system hand over per question?
  6. Is there an interval, and how many runs is the score from?
  7. Can someone else run it again and get the same number?

A score that comes without these answers is a claim about one setup, not a ranking.

How we use this

Trags is an applied AI lab, and testing memory systems is part of our research. Every system we test, our own included, goes through the same pipeline, with the same model writing the answers and a full-context baseline beside it, and every score says whether it was measured or judged.

We have not published results yet. When we do, each score will name the dataset, the answerer, the judge and the budget that produced it.

About Trags