Why the old score was rejected
Snapshot: October 3, 2026. The frozen original execution plan had 349 terminal records: 346 observed completion claims and three privacy-gate infrastructure exclusions. The independent audit found that the old scorer could discard stdout test evidence, treat nonzero setup exits as test failures and skip an unapplied mutation.
The original generation design also cannot be treated as the later apply-and-reveal treatment. Do not infer equivalence merely because both produced completion claims. Source record: docs/swebench-codex-scoring-audit.md.
Corrected instrument and controls
The audited scorer binds patches, plan/scorer hashes, immutable images and sample revisions; proves mutation application; retains named target statuses; and refuses missing, aborted or ambiguous evidence. Real gold/original versus mutant controls exercised three repository paths. A malformed Django mutation was rejected, not relabeled a model failure.
Terminal checkpoints, incomplete adjudication
| Category | Cells | Interpretation |
|---|---|---|
| Adjudicated suite failures | 88 | Named targets and preparation evidence observed. |
| Registry rate limit | 210 | Image retrieval failed before test execution. |
| Other preparation failure | 4 | Requires artifact/baseline inspection. |
| Targets not observed | 29 | No intended-target adjudication. |
| Environment/setup error | 8 | ERROR is not an assertion failure. |
| Aborted runner | 7 | No invented outcome from an abnormal exit. |
| Total | 346 | 88 adjudicated; 258 unadjudicated. Three original privacy exclusions are separate. |
Target success is not whole-suite success
Of the 88 suite failures, 83 had at least one intended target FAILED. Five had all intended targets PASSED while other parsed tests failed. “Zero whole-suite passes” therefore does not mean “no model passed its target tests.”
The unadjudicated infrastructure category is conservative, not a causal acquittal: a model patch can itself break collection or setup. Baseline controls or artifact review are needed before attribution. There is no defensible whole-study success, failure or gaming rate from this coverage.
Preserve failures and keep treatments separate
Repair setup/retrieval causes in a fresh reviewed namespace, retain original records and prove controls before scaling. Report original/treatment and reset/kept-test modes separately. Any post-observation amendment must carry its original failed criterion and revised reading; it is not unchanged preregistration.
This page reports the audited historical pass, not the outcome of ongoing treatment generation or a new scoring run. No private artifacts or live machine paths are published.