Why LongMemEval Scores Are Not Comparable Across Memory Products

LongMemEval is often quoted as if it were a single number. It is not. A published competitor score uses that team's model, judge, packing, and evidence budget. Those choices are part of the result. They are not a same-stack comparison unless the protocol says so.

When we compare memory systems, we keep the reader, judge, question set, and token limit in the same row. A headline without those labels is not enough to rank products.

The reconciled R454 result is 420/500, or 84.0%, graded by GPT 5.5. The later CL1 24k research configuration scored 463/500, or 92.6%, on LongMemEval-S and 1,205/1,540, or 78.25%, on the evaluated LoCoMo categories 1-4.

These figures answer different evaluation questions. R454 and CL1 use different reader pipelines, and CL1 is a development-exposed configuration. A common judge helps make the scoring more consistent, but it does not make every other part of an experiment identical. Product Standard mode uses 8,192 evidence tokens and Deep uses 24,000. Those product modes are not the R454 7,000-token research budget.

Measured Letta and frozen-Zep tables live in the comparison article. Method notes: evaluation methods.

When assessing a comparison, look for the dataset and question count, answering model, prompt and generation settings, evidence-budget definition, grading rubric, and handling of failed answers. A useful report states those conditions alongside the result and identifies any differences between the compared systems.

Each row on the research index has a denominator, reader, evidence budget, judge, and protocol label. Use those results to choose what to test next, then validate on questions representative of your own application. The conversational evaluation is published as Paper 4 on arXiv. These are research measurements under the stated conditions, not product guarantees.