Validate the design, not just the pipeline
A pipeline producing output is not proof that an experiment can distinguish the hypothesis from its alternative. Before scaling, write pass/fail criteria, run a discriminating control through the real pipeline and verify that the intended treatment reached the agent.
An impossible-task study is especially vulnerable to scoring mistakes. If a mutation was never applied, a failure cannot establish the intended impossible condition. If a scorer hides infrastructure failures inside a single “failed” bucket, the denominator can imply certainty that the evidence does not support.
What a positive control actually tells us
A positive control is a case whose expected result is known under the instrument: for example, a solvable task demonstrating that the harness can record success. It checks capability or scoring behavior under that condition. It is not another impossible task and should not be pooled into an impossible-task outcome rate.
A synthetic test-editing patch can provide a different control: it should produce distinguishable outcomes when test edits are kept versus reverted. Those scoring modes answer different questions and must remain separate.
A null result is still a result
The September Codex pilot observed zero false-completion outcomes in both small arms. With no discordant pairs, it did not demonstrate a benefit from the wording change. Zero observed events is not a population rate of zero; the sample remains compatible with materially higher rates.
The scoring audit retains unadjudicated rows instead of turning them into either success or model failure. Registry limits, aborted scoring and target-level errors need explicit treatment. A failed setup can itself have been caused by a patch, so “infrastructure” is a category to investigate, not an automatic acquittal.
Make amendments conspicuous
If a criterion changes after observation, preserve the original criterion, disclose the amendment and report the result under each relevant definition. A study that meets an amended criterion must not be described as meeting unchanged preregistration.
The research pages are dated evidence snapshots, not a live monitor. New results will require a reviewed report with population, exclusions, scoring mode, uncertainty and source provenance. We will publish useful failures and null contrasts alongside positive findings, without exporting private transcripts or credentials.