Marker-Free Benchmarks for Evolving Knowledge: Stop Letting Labels Inflate RAG Scores

Evaluating memory under knowledge evolution is easy to cheat by accident. If a stale fact carries a textual marker — “legacy”, “deprecated”, “[OUTDATED]” — a plain retrieval baseline can disambiguate by reading the label rather than by any temporal mechanism. Scores go up. The science goes down.

We enforce a strict marker-free invariant: in every evolving benchmark, the stale and current versions of a fact are textually identical except for the changed value. No old/new framing. The only signal of currency is ingestion order — which only a temporal mechanism can exploit.

Paper 1 evaluates on two static suites (project-fact QA, multi-session dialogue) and four marker-free evolving suites: code mutation, configuration migration, dependency bumps, and API evolution. We also run a forced-answer supplement that disables abstention on RAG conditions to expose the stale-commitment that polite abstention otherwise hides.

Removing a marker from an earlier contaminated version of a benchmark dropped baseline accuracy by up to 14 points — confirming the effect is real and large enough to invert research conclusions.

If you publish agent-memory results on evolving knowledge:

• State whether your benchmark is marker-free (by test).
• Report stale-fact-error rate separately from accuracy.
• Report forced-answer behavior; abstention can mask structural failure.
• Run competitors on the same local model and harness when claiming architectural wins.

We release harness, prompts, and datasets with the paper so others can reproduce the protocol.