Conversational 0.7.19 is out: read the changelog

What we measured, in plain words

Research results, measured under the stated conditions. Not a service guarantee.

LongMemEval-S, source-aware. The judge read the sources for the same CL1 answers.

95% LongMemEval-S source-aware (475/500), Qwen 3.8 27B Q4

LongMemEval-S, reference-aware. The judge scored those same answers against the benchmark reference.

92.6% LongMemEval-S reference-aware (463/500), Qwen 3.8 27B Q4

LoCoMo, source-aware. Categories 1 to 4. Same retained answers, judged from the sources.

90.91% LoCoMo source-aware (1400/1540), Qwen 3.8 27B Q4

LongMemEval-M, reference-aware. Longer histories. No source-aware score was produced.

85.4% LongMemEval-M reference-aware (427/500), Qwen 3.8 27B Q4

How to read a result

Source-aware accuracy means the judge read the sources behind the answer. Reference-aware accuracy means the judge scored the answer against the benchmark's written reference. Both use reader Qwen 3.8 27B Q4 and judge GPT 5.5. A source-aware number is not a new run of the reader. These are development-exposed research results, not a product mode.

Research

A research record you can examine

Published preprints, unpublished author experiments and development-exposed results stay in separate rows. Every number has a denominator, protocol and source. None of these figures is a product SLA.

Paper 1 · arXiv:2606.26511 Paper 2 · arXiv:2608.20685 Paper 3 · arXiv:2610.07715 Paper 4 · arXiv:2610.05343 Read the blog Reproduce the benchmarks
95%
LME-S source-aware
475/500. Same CL1 answers as the 92.6% reference-aware score. Reader Qwen 3.8 27B Q4. Not a product SLA.
92.6%
LME-S reference-aware
463/500. Judge scored the benchmark reference. Reader Qwen 3.8 27B Q4.
90.91%
LoCoMo source-aware
1400/1540. Same CL1 answers as the 78.25% reference-aware score. Reader Qwen 3.8 27B Q4.
85.4%
LME-M reference-aware
427/500. Longer histories. No source-aware score. Reader Qwen 3.8 27B Q4.
Published

4 papers on arXiv

Paper 1 · Published

1 Published MS-P1-TV-RETRIEVAL arXiv:2606.26511 arXiv preprint · cs.CL / cs.AI

Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge

Deterministic supersession measured against the evaluated retrieval baseline

RAG gives agents access to accumulated knowledge but the evaluated retrieval baseline has no model of time. When a fact changes, cosine similarity surfaces both stale and current values nearly equally. The published study stores facts, then retires contradicted values with a deterministic supersession rule in a bi-temporal ledger. Results belong to the stated benchmarks and protocol, not to a product guarantee. Retrieval systems can add temporal metadata or filters; those designs were not the measured baseline.

  • Cosine AUROC 0.59 for contradiction versus duplicate
  • Evolving knowledge accuracy 0.95 to 1.00 versus RAG 0.20 to 0.47 on the stated local benchmarks
  • Forced-answer stale-fact error near 0 versus RAG 15 to 40 percent in that protocol
arXiv abstract Download PDF Related blog

Paper 2 · Published

2 Published MS-P2-TV-GITHUB arXiv:2608.20685 arXiv preprint · cs.SE / cs.AI

Temporal Validity on Real Software Histories: Eliminating Stale-Fact Errors in Code-Assistant Memory over GitHub Fixes

130 marker-free atomic transitions from 707 real SWE-bench GitHub fixes

The second preprint validates deterministic supersession on real software history. From 707 SWE-bench Lite and Verified GitHub issues, 130 clean atomic state transitions are extracted and rendered marker-free. Scope is explicit: only about 18 percent of real fixes are this clean atomic class.

  • 130 marker-free atomic transitions extracted from 707 real GitHub fixes
  • Answer accuracy 0.91 versus RAG 0.57 to 0.59 on that set
  • Forced-answer stale-fact error about 0 versus RAG 36 to 38 percent in that protocol
arXiv abstract Download PDF Related blog

Paper 3 · Published

3 Published MS-P3-MERGED arXiv:2610.07715 arXiv preprint · cs.SE / cs.CL

When Old Facts Return: Re-Reads, Reverts, and the Limits of Temporal Memory

Combines the former Paper 3 and Paper 4 manuscripts in one published study.

An obsolete value can return when memory encounters the same old statement again. This study examines the ambiguity between rereading a source and a genuine reversal using selected software changes. Supporting experiments consider retired-history exposure and current-source evidence for changed behavior. The findings apply to the retained experiments and their stated judgments, not to universal extraction coverage or a product guarantee.

  • Rereading an old source can reactivate an obsolete value.
  • Blocking reactivation does not by itself identify a genuine reversal.
  • Evidence about the origin of a change is needed to distinguish these cases.
arXiv abstract Download PDF Related blog

Paper 4 · Published

4 Published MS-P4-CONV-27B arXiv:2610.05343 arXiv preprint · cs.CL / cs.IR

MemStrata: 95% and 90.91% Source-Aware Accuracy on LongMemEval-500 and LoCoMo-1540 with a Local Qwen 3.8 27B Q4_K_M Reader

Published arXiv preprint. Results below are author-run research protocols, not product SLAs.

This manuscript is the conversational 27B evaluation record. It is not the synthetic world-knowledge supersession note. R454 is 84.0% (420/500) under GPT 5.5 as judge, with a 7k-token evidence budget. On LongMemEval-S, MemStrata scored higher than Letta at both 8k and 100k token limits in the fresh matched mini protocol and in the Qwen 3.8 27B protocol. On LoCoMo with Qwen 3.8 27B, MemStrata scored higher overall than Letta and frozen Zep. CL1 24k is a later development-exposed configuration. None of these figures is a product SLA.

  • R454: 84.0% (420/500), 7k-token evidence budget, judged by GPT 5.5
  • CL1 24k LongMemEval-S: 92.6% (463/500), author evaluation, different pipeline from R454
  • CL1 24k LoCoMo: 78.25% (1205/1540), author evaluation
  • Fresh matched mini LME500: MemStrata 342/500 vs Letta 8k 299/500 vs Letta 100k 267/500
  • Qwen 3.8 27B LME500: MemStrata CL1 463/500 vs Letta 8k 387/500 vs Letta 100k 438/500
  • Qwen 3.8 27B LoCoMo: MemStrata 7k 1122/1540 vs Letta 8k 1020/1540 vs frozen Zep 8k 913/1540
arXiv abstract Download PDF Related blog

How we evaluate

Each row is one protocol. Result, judge, reader, and token budget stay together. Newer rows link to a public evaluation record.

Benchmark
Status
Comparison

  • Author experiment

    LongMemEval-S accuracy (R454)

    84.0% (420/500)

    Judge
    GPT 5.5, common-judge protocol
    Reader
    Qwen 3.8 27B Q4, high thinking requested, 7,000-token evidence budget
    Dataset
    LongMemEval-S
    Scope
    Author-run R454. Not a product SLA. The 7,000-token evidence budget is the R454/R230 research protocol. Product Standard uses 8,192 evidence tokens and Deep uses 24,000. Historical R454 uses a different reader pipeline from later CL1 work and is not the fresh matched stock control.
    Methods
    Evaluation record

  • Development-exposed

    LongMemEval-S reference-aware accuracy (CL1 24k)

    92.6% reference-aware (463/500)

    Judge
    GPT 5.5, reference-aware (benchmark reference)
    Reader
    Qwen 3.8 27B Q4, CL1 24k research configuration, high thinking
    Dataset
    LongMemEval-S
    Scope
    Reference-aware accuracy. The judge scored the answer against the benchmark reference. The source-aware regrade of these same answers is 475/500 (95%). Development-exposed. Not a product SLA. R454 uses a different reader pipeline.
    Methods
    Evaluation record

  • Development-exposed

    LoCoMo reference-aware accuracy (CL1 24k)

    78.25% reference-aware (1205/1540)

    Judge
    GPT 5.5, reference-aware (benchmark reference)
    Reader
    Qwen 3.8 27B Q4, CL1 24k research configuration, high thinking
    Dataset
    LoCoMo, 1540 scored QA
    Scope
    Reference-aware accuracy. The source-aware regrade of these same answers is 1400/1540 (90.91%). Development-exposed. Not a product SLA. Historical baseline on this population is 1122/1540 = 72.86%. Automated correctness is not human validation.
    Methods
    Evaluation record

  • Author evaluation

    LongMemEval-S source-aware accuracy (CL1 24k)

    95% source-aware (475/500)

    Judge
    GPT 5.5, source-aware regrade
    Reader
    Qwen 3.8 27B Q4, CL1 24k, same retained answers
    Dataset
    LongMemEval-S
    Scope
    Source-aware accuracy. The judge read the sources. Same answers as the reference-aware 463/500 (92.6%): 24 upgrades and 12 downgrades. Not a new reader run. Not a product SLA.
    Methods
    Evaluation record

  • Author evaluation

    LoCoMo source-aware accuracy (CL1 24k)

    90.91% source-aware (1400/1540)

    Judge
    GPT 5.5, source-aware regrade
    Reader
    Qwen 3.8 27B Q4, CL1 24k, same retained answers
    Dataset
    LoCoMo, 1540 scored QA
    Scope
    Source-aware accuracy. The judge read the sources. Same answers as the reference-aware 1205/1540 (78.25%). Not a new reader run. Not a product SLA.
    Methods
    Evaluation record

  • Author evaluation

    LongMemEval-M reference-aware accuracy (CL1)

    85.4% reference-aware (427/500)

    Judge
    GPT 5.5, reference-aware (benchmark reference)
    Reader
    Qwen 3.8 27B Q4, CL1, high thinking
    Dataset
    LongMemEval-M
    Scope
    Reference-aware accuracy. Longer histories than LongMemEval-S. No source-aware score: the judge did not read the full history as source. Reader output failures count as 0. Not a product SLA.
    Methods
    Evaluation record

  • Development-exposed

    Fresh matched mini LongMemEval-S

    MemStrata 68.4% (342/500) vs Letta 8k 59.8% (299/500) vs Letta 100k 53.4% (267/500)

    Judge
    GPT 5.5, common-judge protocol
    Reader
    GPT-4o-mini, fresh matched protocol
    Dataset
    LongMemEval-S
    Scope
    Author evaluation. MemStrata scored higher than Letta at both the 8k and 100k Letta token limits on this protocol. Not a product SLA. Not a write-cost or live-product comparison.
    Methods
    Evaluation record

  • Development-exposed

    Qwen 3.8 27B LongMemEval-S vs Letta

    MemStrata CL1 92.6% (463/500) vs Letta 8k 77.4% (387/500) vs Letta 100k 87.6% (438/500)

    Judge
    GPT 5.5, common-judge protocol
    Reader
    Qwen 3.8 27B, high thinking. Letta arms at 8k and 100k token limits.
    Dataset
    LongMemEval-S
    Scope
    Author evaluation. MemStrata CL1 24k scored higher than Letta at both token limits. Letta 100k actual evidence was below a binding 100k ceiling. Not a product SLA. R454 84.0% is a different pipeline.
    Methods
    Evaluation record

  • Development-exposed

    Qwen 3.8 27B LoCoMo vs Letta and frozen Zep

    MemStrata 7k 72.86% (1122/1540) vs Letta 8k 66.23% (1020/1540) vs frozen Zep 8k 59.29% (913/1540)

    Judge
    GPT 5.5, common-judge protocol
    Reader
    Qwen 3.8 27B, high thinking. MemStrata 7k-token evidence budget. Letta and frozen Zep 8k-token limits.
    Dataset
    LoCoMo, 1540 scored QA
    Scope
    Author evaluation. MemStrata used a 7k-token evidence budget. Letta and frozen Zep used 8k-token limits. MemStrata scored higher overall. Letta led multi-hop on this protocol. Frozen Zep is a frozen snapshot, not a live product. Not a product SLA. Later CL1 24k MemStrata is 1205/1540 on the same population and is a different configuration.
    Methods
    Evaluation record
Metric Result Judge Reader Dataset Class Scope Methods
LongMemEval-S accuracy (R454) 84.0% (420/500) GPT 5.5, common-judge protocol Qwen 3.8 27B Q4, high thinking requested, 7,000-token evidence budget LongMemEval-S Author experiment Author-run R454. Not a product SLA. The 7,000-token evidence budget is the R454/R230 research protocol. Product Standard uses 8,192 evidence tokens and Deep uses 24,000. Historical R454 uses a different reader pipeline from later CL1 work and is not the fresh matched stock control. Record
LongMemEval-S reference-aware accuracy (CL1 24k) 92.6% reference-aware (463/500) GPT 5.5, reference-aware (benchmark reference) Qwen 3.8 27B Q4, CL1 24k research configuration, high thinking LongMemEval-S Development-exposed Reference-aware accuracy. The judge scored the answer against the benchmark reference. The source-aware regrade of these same answers is 475/500 (95%). Development-exposed. Not a product SLA. R454 uses a different reader pipeline. Record
LoCoMo reference-aware accuracy (CL1 24k) 78.25% reference-aware (1205/1540) GPT 5.5, reference-aware (benchmark reference) Qwen 3.8 27B Q4, CL1 24k research configuration, high thinking LoCoMo, 1540 scored QA Development-exposed Reference-aware accuracy. The source-aware regrade of these same answers is 1400/1540 (90.91%). Development-exposed. Not a product SLA. Historical baseline on this population is 1122/1540 = 72.86%. Automated correctness is not human validation. Record
LongMemEval-S source-aware accuracy (CL1 24k) 95% source-aware (475/500) GPT 5.5, source-aware regrade Qwen 3.8 27B Q4, CL1 24k, same retained answers LongMemEval-S Author evaluation Source-aware accuracy. The judge read the sources. Same answers as the reference-aware 463/500 (92.6%): 24 upgrades and 12 downgrades. Not a new reader run. Not a product SLA. Record
LoCoMo source-aware accuracy (CL1 24k) 90.91% source-aware (1400/1540) GPT 5.5, source-aware regrade Qwen 3.8 27B Q4, CL1 24k, same retained answers LoCoMo, 1540 scored QA Author evaluation Source-aware accuracy. The judge read the sources. Same answers as the reference-aware 1205/1540 (78.25%). Not a new reader run. Not a product SLA. Record
LongMemEval-M reference-aware accuracy (CL1) 85.4% reference-aware (427/500) GPT 5.5, reference-aware (benchmark reference) Qwen 3.8 27B Q4, CL1, high thinking LongMemEval-M Author evaluation Reference-aware accuracy. Longer histories than LongMemEval-S. No source-aware score: the judge did not read the full history as source. Reader output failures count as 0. Not a product SLA. Record
Fresh matched mini LongMemEval-S MemStrata 68.4% (342/500) vs Letta 8k 59.8% (299/500) vs Letta 100k 53.4% (267/500) GPT 5.5, common-judge protocol GPT-4o-mini, fresh matched protocol LongMemEval-S Development-exposed Author evaluation. MemStrata scored higher than Letta at both the 8k and 100k Letta token limits on this protocol. Not a product SLA. Not a write-cost or live-product comparison. Record
Qwen 3.8 27B LongMemEval-S vs Letta MemStrata CL1 92.6% (463/500) vs Letta 8k 77.4% (387/500) vs Letta 100k 87.6% (438/500) GPT 5.5, common-judge protocol Qwen 3.8 27B, high thinking. Letta arms at 8k and 100k token limits. LongMemEval-S Development-exposed Author evaluation. MemStrata CL1 24k scored higher than Letta at both token limits. Letta 100k actual evidence was below a binding 100k ceiling. Not a product SLA. R454 84.0% is a different pipeline. Record
Qwen 3.8 27B LoCoMo vs Letta and frozen Zep MemStrata 7k 72.86% (1122/1540) vs Letta 8k 66.23% (1020/1540) vs frozen Zep 8k 59.29% (913/1540) GPT 5.5, common-judge protocol Qwen 3.8 27B, high thinking. MemStrata 7k-token evidence budget. Letta and frozen Zep 8k-token limits. LoCoMo, 1540 scored QA Development-exposed Author evaluation. MemStrata used a 7k-token evidence budget. Letta and frozen Zep used 8k-token limits. MemStrata scored higher overall. Letta led multi-hop on this protocol. Frozen Zep is a frozen snapshot, not a live product. Not a product SLA. Later CL1 24k MemStrata is 1205/1540 on the same population and is a different configuration. Record

This table is a versioned results registry, not a product guarantee. Published preprints, unpublished author experiments and development-exposed configurations stay in separate rows. R454 is 84.0% (420/500) judged by GPT 5.5, with a 7k-token evidence budget. On LongMemEval-S, MemStrata scored higher than Letta at both the 8k and 100k token limits in the fresh matched mini protocol and in the Qwen 3.8 27B protocol. On LoCoMo with Qwen 3.8 27B, MemStrata at a 7k-token evidence budget scored higher overall than Letta 8k and frozen Zep 8k. CL1 24k later scored 463/500 = 92.6% on LongMemEval-S and 1205/1540 = 78.25% on LoCoMo as author evaluations. R454 and CL1 use different reader pipelines. Each row is one protocol, not a combined system score. These are author-run protocols, not live product or write-cost comparisons. No Mem0 or Graphiti comparison is published here.

Citations: Yadav, N. (2026). Temporal Validity in Retrieval Memory. arXiv:2606.26511. Yadav, N. (2026). Temporal Validity on Real Software Histories. arXiv:2608.20685. Yadav, N. (2026). When Old Facts Return: Re-Reads, Reverts, and the Limits of Temporal Memory. arXiv:2610.07715. Yadav, N. (2026). MemStrata: 95% and 90.91% Source-Aware Accuracy on LongMemEval-500 and LoCoMo-1540 with a Local Qwen 3.8 27B Q4_K_M Reader. arXiv:2610.05343.

From the blog

Explainers & deep dives

All posts →

2026-09-19 · 5 min

Standard versus Deep Evidence: How to Choose the Context Budget

2026-09-19 · 6 min

Local AI Memory: What Stays on Your Device and What Goes to a Model Provider

2026-09-19 · 6 min

Why an AI Answer Can Cite a Source and Still Be Wrong