Pre-register a Benchmark Before You Run It

Every rule below exists because a reviewer caught us breaking it. The round is named each time.

Most advice about evaluating AI systems is about what to measure. This page is about something narrower and, in our experience, more decisive: fixing the method in place before the numbers exist, so that when the results turn out inconvenient you cannot quietly adjust the question.

We did not learn this from the literature. We learned it by preparing an evaluation for publication and being blocked five times in a row by independent review — and then cutting it. Each round found a different way for a plausible-looking result to outrun its evidence. The counts, transcribed from our own trace:

Round	Verdict	Findings	The defect that mattered
R1	BLOCKED	19 P0 / 3 P1 / 1 P2	A ranking-style benchmark whose inputs, scores and method history were not evidenced
R2	BLOCKED	8 P0 / 3 P1	The validity floor was written after we knew which runs it would exclude
R3	BLOCKED	9 P0 / 3 P1	The headline was retracted; the data layer was not
R4	BLOCKED	A7 / B4 / C2	A failed prerequisite metric serialised the downstream metrics as zero
R5	BLOCKED	A6 / B3 / C1 · A3 / B8 / C1	Per-product conclusions still alive in run logs; the measuring instrument recorded no version

(R1–R3 were counted in P0/P1/P2 severities; from R4 the reviewer switched to an A/B/C convergence classification, so those are not P0 counts and should not be compared as if they were. R5 was two reviews in parallel — see rule 1.)

Who this is for — and who should skip it

For you if: you intend to publish a number that compares systems, or that someone will act on; and you can afford one reviewer who did not do the work.

Skip it if: you are exploring. Exploration should be cheap, fast and undeclared — the cost of this discipline is only worth paying at the moment you decide the result will leave the building. Deciding that afterwards is exactly the failure mode preregistration prevents, so decide early and err toward pre-registering.

The six rules

Rule 1 — The person who writes the method must not be the person who runs it

From R2. Our validity floor — the minimum number of successful runs required for a case to count — was added after the fact, and it determined which cases were reported as having no result. It was written with the exclusion list already visible. Nothing about it was dishonest; it simply could not be shown to be independent, which for a published number is the same problem.

The structural fix is cheap: separate the roles. In the worked instance below, the method was written by the orchestrating session and executed by a different agent, which was recorded in the method itself as a deliberate choice. Someone who is not running the experiment cannot tune the method to flatter the results, because they have not seen any.

The same logic applies one layer up, at review. Our reviews ran on a different model family from the one that produced the work — a producer cannot certify itself. Round five ran two reviewers in parallel, and the result is the strongest methodological finding we have:

Seven A-class findings after de-duplication. Only two overlapped. One reviewer alone found four the other missed; the other found one nobody else saw. Both sets were A-class.

Single-reviewer recall on defects that spread across several layers is visibly incomplete. A second reviewer bought more than a second round with the same reviewer would have.

Rule 2 — Freeze the method in a form a stranger can verify

"We wrote the method first" is a claim about the past that a reader has no way to check. Make it checkable: hash the frozen region and publish the hash.

Our method document is frozen in sections 1–10 (lines 1–122). Those lines hash identically before and after the run:

head -n 122 method.md | md5sum
edc553bc3dca31a760c2c109dea9a9c5     # before the run
edc553bc3dca31a760c2c109dea9a9c5     # after the run

Note what is being hashed. The whole file hash did change — deliberately — because everything below line 122 is an append-only deviation log. A frozen document that can never be appended to is a document people quietly edit anyway. Freeze the region that constitutes the promise; keep a growing record beneath it; hash the region, not the file. Claiming the file was unchanged would be false, and trivially disprovable.

Rule 3 — Declare the validity floor and the stop-loss, and say they bind this run

Two things must be written before the first measurement:

• The validity floor — the controls that must pass for the run to count at all. Ours were four: the environment answers normally, the pre-existing rules are uncontaminated, a genuine 404 is a genuine 404, and the baseline is unchanged when the test material is removed. Any control failing voids the entire run; the data is not published, and the attempt is recorded as did-not-complete.
• The stop-loss — when to abandon. Ours: no valid response within five minutes, stop; three requests producing three different results, record that cell as not observable and do not widen the sample until it stabilises, which is the mechanised form of not p-hacking; thirty minutes total, stop and record what exists.

The clause that makes these real is one sentence long: "this rule applies to the present run." Without it, a stop-loss written during a run becomes a tool for selecting outcomes after the fact — which is precisely the R2 defect, wearing safety equipment.

Rule 4 — "Could not be measured" is not a score of zero

From R4. In the earlier evaluation, one metric was a prerequisite for the four after it. When it failed, the remaining four could not be observed at all — and the schema had nowhere to put that, so they were serialised as 0. A reader, and every downstream table, then saw four terrible scores where there had been no measurement.

The fix is in the data model, not the prose: three values, not two.

Value	Means
effective	Behaved as specified
silently-ignored	Did not work and produced no error signal
not-observable	Preconditions unmet — this cell has no measurement

Two further disciplines keep the third value honest. Define the mapping before the run — which observation lands in which bucket, written down while the buckets are still empty. And report the count even when it is zero: our worked run recorded not-observable = 0, and said so explicitly, because "nothing was unmeasurable" is a result, while a silently absent category is just an absence.

Rule 5 — When you retract a conclusion, sweep the evidence layer too

From R3, and again from R5. After the ranking claim was withdrawn, the headline was gone — and the median-scores table was still shipping, merely renamed. A per-product conclusion that has been retracted at the title layer and left standing in the data layer has not been retracted.

Round five found the same shape one level deeper: named product conclusions still alive inside internal run logs, all of them flattering to us, long after the page itself had stopped making them.

There is a mechanical lesson attached, and it cost us a live defect. When the evaluation was finally cut, the commit that removed it stated that dangling references had been cleared. It had grepped the route. It had not grepped the name. A day later, two promises pointing at the deleted work were still being served on the production page — found with a plain curl, not a reasoning step.

Grep the topic's name, not just its URL. A route belongs to the thing you deleted. Its name is scattered across everything that ever referred to it.

Rule 6 — The instrument must record its own version

From R5's second reviewer. One metric was defined as "open the artifact in a second application." The external run recorded that application's exact build hash. Our own three runs logged a bare soffice command — no version, nowhere. The page nonetheless described the check as having been performed with a specific LibreOffice version, and the log could not support that sentence. The receiver is part of the measuring instrument; an unversioned receiver makes the measurement unrepeatable.

The same rule turned on our review process itself, and it is the most uncomfortable finding in this whole page. None of the first five review reports recorded which model or command produced them — a search for any of the tool names across all five returned nothing. Because of that gap, our own first written account of the review chain got the mechanism wrong: it inferred the reviewer's identity from the prose style of the reports. The correction came from reading the original dispatch records.

If the entire value of a review is "an independent party looked at this," then who looked is the finding — and it has to be written down at the time, by the process, not reconstructed later from tone.

Every report now carries a provenance header: model, dispatch command, baseline commit, verdict, remediation commit.

A worked instance

The most recent study run under these rules was a small one: which redirect rule forms fail without warning on a static host. It is a good demonstration precisely because it is unglamorous.

Discipline	How it showed up
Rule 1	Method written by one session, executed by a different agent
Rule 2	Sections 1–10 hashed edc553bc… before and after
Rule 3	Four controls, all passed; three stop-loss rules, none triggered — first control answered in ~2 s against a 300 s limit, total run ~17 s against a 30 min limit
Rule 4	Three-value schema; not-observable finished at 0 and the zero was reported
Rule 5	Scope limit written into the method up front: local simulator, no claim about production
Rule 6	Tool version, node version, script hashes and the frozen input snapshot fingerprint all recorded

Ten deviations, D1 through D10, are logged below the frozen line. Two of them are the point of the whole exercise:

• D5 — the method was wrong, and we did not fix it retroactively. Mid-run we found that our own reading table conflated "reached the cross-origin target" with "returned local content." Two cells had been scored effective under that flawed rule. We left the verdicts as the frozen rule produced them, recorded the true observation alongside, and noted that a future version of the method needs a provenance test. Rewriting the rule to produce the answer we now believed would have destroyed everything preregistration was protecting.
• D8 — an aborted first attempt, and why the rerun was legitimate. A shell bug meant the final rule was never actually requested. This was found before any result code had been looked at, so restarting was not a result-dependent choice — and the raw output of the aborted attempt was kept. A rerun after seeing the numbers would have been a different act entirely.

What this does and does not buy you

• Verified: the six rules each trace to a named round in our own review history; the counts above are transcribed from that record. The worked instance really ran — frozen method, hashed region, four controls passed, 144 requests, deviations D1–D10 logged.
• Not verified: that these rules generalise. This is one project's history, not a study of preregistration's effects.
• Explicitly not claimed: the worked instance has not been through an independent model-family review. It demonstrates the discipline; it is not certified by it.
• The honest bottom line: the evaluation that generated these rules was worked from a preregistered method — and was still withdrawn. Both final verdicts were BLOCKED, and review was stopped by human decision rather than by passing. Stopping review is not approval. Preregistration bounded how far a wrong conclusion could travel. It did not make the conclusion right.

Limitations

One project, five rounds, no control group. Rules 1 and 6 assume a second reviewer and a second executor you may not have; rules 2 through 5 work solo. And passing all six earns you a defensible method, not a useful result — ours was defensible and got cut anyway.

Take the assets

Asset	What it is
Method template	The ten frozen sections, with the clauses that are easiest to leave out
Pre-flight checklist	18 binary gates to clear before the first measurement
Deviation log template	The append-only format, with the before/during/after distinction that makes it credible

The next action

Take the method template and fill in section 3 — fixed inputs and sample size — for the next evaluation you were going to run informally. Then hash sections 1 through 10 and paste the hash into your run log before you start. That single command is the difference between a method you followed and a method you can prove you followed.