The Value-Change Ceiling: Only ~19% of Real Code Fixes Are Clean Supersessions
Across 707 SWE-bench fixes, deterministic temporal memory natively covers ~19% of evolution. Contract extraction lifts to ~24%. The remaining ~76% is logic — and overclaiming extraction is a research risk.
By Neeraj Yadav. MemStrata is built by Neeraj Yadav.
Scope is a first-class research result
After Papers 1–2, the natural question is not “does supersession work?” but “on how much of real code evolution does it apply?” The earlier taxonomy work examined 707 real GitHub fixes (SWE-bench). The merged Paper 3 treats this extraction study as exploratory scope context, not a universal coverage estimate.
Headline: ~19% native value-change ceiling. A bounded contract extractor (params, arity, return type) lifts the extraction ceiling to only ~24% overall. Full-file AST extraction across declined fixes recovers just 6.6% additional contract changes — not a flood of hidden triples.
The majority is logic — and that is okay
About 76% of fixes are logic/control-flow that do not reduce to a clean (subject, relation, object) triple. Pretending otherwise would inflate claims and corrupt the product. MemStrata’s research stance is deliberate: own the CERTAIN value-change spine perfectly, then handle the rest with the earlier detect-and-flag design rather than hallucinated structure.
Human validation on a stratified 82-fix sample (Cohen’s κ = 0.144 on fine 5-way labels) shows disagreement concentrates off the load-bearing axis: the native value rate remains stable even when membership shifts. Two bounds survive: the ~19% ceiling and logic majority.
Recency is not a temporal model
Controlled experiments show recency-aware RAG — including LLM “pick the newest chunk” rerankers — fails to match the ledger. A recency-adversarial re-read of an old value exposes a sharp boundary where both recency and a naive ledger can resurrect stale state without additional guards.
For teams evaluating memory products: ask for taxonomy coverage, not just a single accuracy number on cherry-picked value flips.
Previous: SWE-bench Longitudinal: How to Build an Evolving-Code Evaluation Next: Detect-and-Flag: Honest Memory for the 76% of Code That Is Not a Triple