We killed our king-selection proxy — and rebuilt it
A subnet that crowns recipes by a metric uncorrelated with real quality is selecting for noise. So we pre-registered a test of our own selection gate — kill thresholds set before any data landed — and ran it.
The verdict
The 20-step proxy said R04 (4× learning rate) had the lowest loss of all 12 recipes; at 1000 steps it had the highest — it diverges. The current rule would have crowned it. Meanwhile R12 (QK-norm) was middle-of-pack to the proxy but ranked first at ground truth — the recipe most worth canonizing was invisible to the gate. Our own selection function would have made it harder to find the best recipe in our set.
Why pre-register the kill
We set the thresholds before any data landed: keep the proxy if Spearman ρ ≥ 0.70, kill it if ρ < 0.30, with the pre-registration hash committed before any spend. The data came back at ρ = 0.203 — below the kill line, with no post-hoc rationalization available. A subnet that crowns recipes by a metric uncorrelated with real quality is selecting for noise, and we’d rather find that out for $3.09 than canonize bad recipes expensively.
What replaced it
We moved king selection to a cross-scale downstream ladder: each submission trains at three scales — a ~30-second pre-screen, a few-minute checkpoint, and a ~124M-parameter run scored against a downstream-task ensemble — with val-BPB kept as a cheap no-regression check. 124M is the scale the open pretraining community has coordinated on, and the scale where DCLM shows recipe rankings start to transfer toward frontier scale. We’d rather be wrong cheaply than canonize bad recipes expensively. This is the same selection criterion the transfer-credibility test now puts under the microscope.
The thread this opened
Moving to a cross-scale downstream ladder raised the obvious next question: does the cheap rung actually predict quality at the expensive scale? That question is exactly what the transfer-credibility test now measures — the same selection criterion, under the microscope.