SWE-bench Longitudinal: How to Build an Evolving-Code Evaluation

Paper 1 proves the mechanism on carefully constructed evolving benchmarks. Paper 2 asks whether temporal validity still holds on real software history.

The publication overview introduces the Paper 2 findings. This article explains how the evolving-code evaluation is built and what its scope allows you to conclude.

This evaluation uses 130 clean atomic transitions extracted from 707 SWE-bench GitHub fixes. Each scenario presents an earlier state and a later change, then asks about the current value. The material does not label the old passage as outdated or the new one as the answer.

That construction tests whether a memory system can select the applicable fact when both versions remain available. It also limits the claim: clean value changes are a subset of software evolution. A successful answer on this set does not establish that the system understands arbitrary changes to control flow, architecture, or behavior.

For the exact measurements and evaluation settings, read the published Paper 2. Those figures belong to that paper's protocol. They are not a LongMemEval product score.

Suppose a fictional config file said timeout_seconds = 30 on Monday, then timeout_seconds = 90 on Tuesday. Keep both notes. Do not add labels such as OLD or NEW.

• Current-state question: What is the timeout now? The supported answer is 90, citing Tuesday.
• Historical question: What was the timeout before Tuesday? The supported answer is 30, citing Monday.

A system can answer the current question correctly while discarding the earlier record. That is a different failure from serving Tuesday's value as if it were still Monday's. Explicit old/new cues in the test text make the task easier than a marker-free protocol, because the label itself reveals currency.

A stale-fact error is serving a superseded value as current. An unrelated incorrect answer, such as inventing 60, is a different error class. Paper 2 reports both accuracy and stale-fact error on the atomic-transition subset. Only about 18 percent of the 707 real fixes are this clean atomic class.