How these results were produced

Shared scoring rules

• These conversational evaluations were run by the author, using one automated judge model, GPT 5.5. Treatment labels were hidden from the grader; that does not make the evaluation independent.
• CL1 was developed using these benchmark families. These are development-exposed results, not a held-out customer-task evaluation or an independent replication.
• LongMemEval-S population: 500 questions. LoCoMo population: 1,540 scored questions in categories 1-4. Category 5 is excluded.
• Evidence tokens are the retrieved-source budget packed for the reader. They do not include system instructions, the question, later conversation, or output capacity. Tokenizers are not interchangeable across research budgets and product modes.
• Reuse policy: a saved answer is reused only when the complete request and raw response match. Changed evidence orderings are not reused.
• Empty or missing final answers count as incorrect. Two MemStrata LoCoMo baseline items are recorded as same-seed non-answers at zero.
• Automated GPT-5.5 scores are not human validation. Abstention correctness is not an all-answer fabrication rate.
• Product Standard mode uses 8,192 evidence tokens. Product Deep uses 24,000. Those installed modes are not the research budgets below.

R454 LongMemEval-S

84.0% (420/500). Run identifier R454. Verified 2026-09-12.

• Dataset: LongMemEval-S, 500 questions.
• Reader: Qwen 3.8 27B Q4, high thinking requested.
• Evidence budget: 7,000 tokens. That is the R454/R230 research protocol, not product Standard or Deep.
• Judge: GPT 5.5. Original local-judge score is not the public figure.
• Pipeline: historical R454 reader pipeline. It is not the later CL1 pipeline and is not the fresh matched stock control.
• Development exposure: author-run experiment, not a shipping measurement.

CL1 24k LongMemEval-S

92.6% reference-aware (463/500). Run identifier CL1 LME500. Verified 2026-09-13.

• Dataset: LongMemEval-S, 500 questions.
• Reader: Qwen 3.8 27B, high thinking, CL1 24k research configuration.
• Evidence budget: 24,000-token research admission. Observed packets can sit below that ceiling.
• Judge: GPT 5.5, same common-judge protocol as R454.
• Development exposure: CL1 was developed using these benchmark families. This is not a held-out customer-task score.
• R454 and CL1 are separate pipelines. The 8.6-point numeric gap does not isolate a single retrieval change.

CL1 24k LoCoMo

78.25% reference-aware (1205/1540). Run identifier CL1 LoCoMo. Verified 2026-09-14.

• Dataset: LoCoMo, 1,540 scored questions (categories 1-4). Category 5 excluded.
• Reader: Qwen 3.8 27B, high thinking, CL1 24k research configuration.
• Judge: GPT 5.5.
• Historical baseline on this population is 1,122/1,540 = 72.86% at a 7k-token evidence budget. That baseline is a different configuration.
• Development-exposed. Automated correctness is not human validation or token F1.

Source-aware regrades of the same CL1 answers

These are changes to the judge's evidence, not new reader runs or independently replicated improvements.

• LongMemEval-S: 95% source-aware (475/500), compared with 92.6% reference-aware (463/500) on the same answers. GPT 5.5 read the sources; 24 judgments were upgraded and 12 downgraded. Regrade recorded 2026-09-22.
• LoCoMo: 90.91% source-aware (1400/1540), compared with 78.25% reference-aware (1205/1540) on the same answers. Categories 1–4; category 5 excluded. Regrade recorded 2026-09-23.
• Both are author-run, automated, development-exposed evaluations. They do not establish human-validated accuracy, held-out generalization, or a product guarantee.

CL1 LongMemEval-M

85.4% reference-aware (427/500). Verified 2026-09-27.

500 questions: 179/200 in the first sample plus 248/300 in the remainder. Reader: Qwen 3.8 27B Q4. Judge: GPT 5.5 against the benchmark reference. There is no source-aware score: the judge did not read the full longer histories. Reader output failures remain incorrect. This is an author evaluation, not a product SLA.

Fresh matched mini LongMemEval-S

MemStrata 68.4% (342/500) vs Letta 8k 59.8% (299/500) vs Letta 100k 53.4% (267/500). Verified 2026-09-12.

• Dataset: LongMemEval-S, 500 questions.
• Reader: GPT-4o-mini, fresh matched protocol.
• MemStrata: nominal 7,000-token retrieval budget. This is not an equal-evidence-budget comparison with the configured Letta limits below.
• Letta arms: configured 8k token limit and configured 100k token limit. Configured limits are not observed evidence-token usage.
• Judge: GPT 5.5.
• MemStrata scored higher than Letta at both configured Letta limits on this protocol. Not a write-cost or live-product comparison.

Qwen 3.8 27B LongMemEval-S versus Letta

MemStrata CL1 92.6% (463/500) vs Letta 8k 77.4% (387/500) vs Letta 100k 87.6% (438/500). Verified 2026-09-13.

• Dataset: LongMemEval-S, 500 questions.
• Reader: Qwen 3.8 27B, high thinking, for the MemStrata CL1 arm and the Letta Qwen arms.
• MemStrata CL1 uses a 24,000-token evidence budget. Configured ceilings and observed usage differ across arms; this does not isolate memory-system quality at equal cost or equal budgets.
• Letta configured limits: 8k and 100k. Letta 100k actual evidence had mean 23,878.16 tokens, median 24,036, range 14,134 to 29,011. The 100k ceiling was not binding.
• Judge: GPT 5.5.
• MemStrata CL1 24k scored higher than Letta at both configured limits. Not a product SLA.

Qwen 3.8 27B LoCoMo versus Letta and frozen Zep

MemStrata 7k 72.86% (1122/1540) vs Letta 8k 66.23% (1020/1540) vs frozen Zep 8k 59.29% (913/1540). Verified 2026-09-12.

• Dataset: LoCoMo, 1,540 scored questions (categories 1-4).
• Reader: Qwen 3.8 27B, high thinking.
• MemStrata evidence budget: 7,000 tokens. Letta and frozen Zep: configured 8k token limits. Those limits are not equal-budget proof.
• Frozen Zep is a frozen snapshot, not the current live service. Original wire receipts are incomplete, so exact request parity is limited.
• MemStrata scored higher overall. Letta led multi-hop on this protocol. Later CL1 24k MemStrata is 1,205/1,540 on the same population and is a different configuration.