Open verification · Version 0.3.0
Three research experiments, with fixed question sets, prepared evidence, inspectable prompts and recorded grading rules. Download the sealed bundles and use your own compute to test the reported results.
These are author results on development-exposed benchmarks, not product guarantees or independently replicated scores. The harness replays the reader on frozen retrieval outputs; it does not regenerate MemStrata’s proprietary index or retrieval process.
The figures below are saved reference scores under automated GPT-5.5 grading.
92.6%
463 / 500 correct · reference result
Qwen 3.8 27B · Q4_K_M · high thinking
24,000 evidence tokens
All 500 questions. Fixed reader requests and automated GPT-5.5 judgments.
Download bundle · 16.3 MBData license: MIT
16,267,573 bytes
c5c04a74346110d627aad79acd99ed2b0b2600858c4ba2ed4a9df36613be61ee Lane: cl1-lme500-24k-v1
78.25%
1,205 / 1,540 correct · reference result
Qwen 3.8 27B · Q4_K_M · high thinking
24,000 evidence tokens
Categories 1–4. Four primary output failures remain zero. No recovery substitution.
Download bundle · 31.4 MBData license: CC-BY-NC-4.0
31,422,135 bytes
d448be21972a4d573ffc12786eaa4fcf8880957237e2297ca538b13069f5d4f5 Lane: cl1-locomo1540-24k-v1
68.4%
342 / 500 correct · reference result
GPT-4o-mini-2024-07-18 · fixed routed reader
Nominal 7,000-token retrieval budget
September 9 retained run. This is a separate experiment, not R230 or R454.
Download bundle · 11.2 MBData license: MIT
11,216,878 bytes
2cd6aa984ae90fdcdc261ca18ba7631e5faf6ac4d989a98a483d0ef3c4b9a154 Lane: mini-lme500-7k-20260909-v1
LoCoMo here uses binary correctness on 1,540 questions across ten conversation histories, not token F1. Its data is distributed separately for non-commercial benchmark research under CC BY-NC 4.0. It is not part of MemStrata’s commercial product or the harness wheel. Attribution: Maharana et al., Snap Research. LongMemEval: Di Wu et al., MIT license. Neither project endorses MemStrata.
Check publisher signatures, every file hash, question membership, saved responses and score aggregation. Free and offline. This confirms the retained result; it is not fresh inference.
Use the exact local Qwen weights or pinned mini snapshot, evidence, prompts and settings. Reader inputs do not include evaluator-only gold answers. Each attempt is retained.
Apply the recorded GPT-5.5 xhigh rubric to new answers. Report runtime differences and all failures. An incomplete run gets no overall accuracy; a valid lower result is not rejected.
Use a fresh Python environment. Download the LME500 ZIP above, then:
python -m pip install "git+https://github.com/yadu9989/memstrata-lme500-harness.git@v0.3.0"
memstrata-reproduce unpack cl1-lme500-24k-v1 cl1-lme500-24k-v1.zip bundles/cl1-lme500
memstrata-reproduce verify bundles/cl1-lme500See the full protocol for Windows/macOS/Linux commands, Qwen model-tag restoration, paid mini calls, judge setup and resume rules. Vercel hosts this documentation; it does not run your benchmark models.
Paper numbers are historical labels, not a count of publications. The current registry lists four published arXiv preprints. Older harness text calls the conversational R454 work “Paper 5”; that is the earlier internal numbering. The conversational preprint is now arXiv:2610.05343 (Paper 4 in the current registry), after earlier manuscripts were consolidated. Use arXiv IDs and versioned experiment IDs when comparing records.
Auditors need to inspect the actual reader instructions. Local encryption or encoding would not keep those instructions secret from the person executing them. The proprietary retrieval implementation stays private.
Signed manifests and pinned hashes identify the released inputs. They cannot stop someone controlling a computer from modifying code, reading public gold, fabricating receipts or publishing a false score. Independent execution and provider records matter. A lower result under the agreed protocol deserves investigation.
All saved populations are verified offline. Live smoke tests are small transport checks, not new full-benchmark scores. There is no measured ±X-point rerun guarantee yet. Fixed seeds do not establish one, and a dataset confidence interval is not repeat-run variability.
The original grader used GPT-5.5 xhigh through Codex CLI 0.153.4. A hosted server snapshot is not independently attested. Exact-binary mode checks the original binary hash; a compatible newer runtime must be explicitly selected and is labeled in the report. The harness never silently switches graders.
R230 and R454 have 497 matching context hashes out of 500, but lack a complete original request capture. This release uses the fully retained September 9 mini run. R454’s common GPT-5.5 regrade remains 84.0% (420/500); it is not relabeled as the mini experiment.
Read the historical audit