What the experiments can establish
Commandem studies agent completion, verification, context and orchestration behavior. A result is tied to a specific instrument, provider, model, task distribution and treatment. Evidence from one setup does not establish a universal model effect.
Freeze criteria and demonstrate that the instrument can discriminate.
Prove the intended condition reached the agent from retained logs.
Score independently; keep pending, infrastructure and exclusions separate.
Current working manuscript
Completion claims and verifier integrity in coding agents
Jonathan Scarpelli and Ayesha Akhter
Read the 13-page manuscript on overlapping completion/subversion signals, turn-limit interpretation, task coverage and historical-binary confounds. Includes the failed replication and a proposed, unrun follow-up.
Reviewed snapshots
Codex completion-wording replication
Two impossible-task arms each observed zero false completions in ten cells. Small samples and a separately frozen harness limit the contrast.
SWE-bench independent-scoring audit
346 checkpoints contain 88 adjudicated suite failures and 258 unadjudicated rows. This is not a defensible whole-study failure or gaming rate.
How to read the homepage figures
The homepage's September archive snapshot reports 4,370 indexed runs, 95 favorable versus zero harmful discordant matched tasks, and 25 rejected completion claims in sampled product telemetry. Those are three different populations, not interchangeable denominators. Runner versions differed in the matched study; the count alone does not identify a causal effect. Rejected claims in sampled telemetry are not a population rate.
These legacy figures remain dated descriptive observations. The reports here do not silently update or pool them with newer treatment results. Publication-grade estimates need study-specific methods, uncertainty and complete adjudication.
Ongoing work is not a final result
Treatment generation, model claims and scoring checkpoints are different stages. This static page is not a live experiment monitor. Unfinished or unscored cells do not establish success, failure or gaming. Reset-tests and kept-tests modes, original and treatment arms, and protocol amendments stay separate.
Reproducibility and privacy
Before a full run, write pass/fail criteria, include controls with different correct outcomes, verify treatment delivery and obtain owner approval for resource use. Record frozen runner/scorer/input hashes, environment revisions, exclusions and failed attempts. Retained private logs are not automatically publishable data.
Public-safe methods and analysis excerpts appear in the linked reports. Full private roots, transcripts, credentials and dataset redistribution are not exposed by this website. See the research interpretation article and Privacy.