Does a cheap attested gate predict recipe quality at scale?
Everyone can now run the LLM-pretraining recipe search cheaply. Almost nobody proves their cheap winner actually holds at larger scale — and nobody binds that proof to a run you can verify happened. This is Ralph’s pre-registered test of exactly that, published either way.
The question
Recursive, Prime Intellect and ScaleAutoResearch made the recipe search common. The durable asset isn’t the search — it’s a cheap, attested gate whose verdicts you can trust, and the lineage of proven improvements it produces. Call the white space DataDecide-under-attestation: calibrate that cheap gate’s predictive validity, then publish it. The protocol’s credibility is downstream of this being falsifiable, so we don’t assume it; we measure it.
The test
The gate is held-out validation BPB at 124M parameters (NanoGPT-Speedrun scale). The reference is the same recipes retrained at larger scale on the identical metric. Transfer is the Spearman ρ — and the pairwise decision-accuracy — between the two rankings, across ~24 config-flag-gated interventions (optimizer, schedule, positional encoding, activation, QK-norm, weight decay, batch/seq/LR sweeps), plus must-die probes like an 8×-too-high learning rate. Attestation is the protocol’s half: a score binds to execution on confidential-compute hardware, so a number maps to a run that provably happened, not one a miner reported. This calibration itself runs on ordinary GPUs — what it tests is whether the cheap gate that attested protocol leans on actually predicts at scale.
Measuring cheap first paid off
Our planned gate — downstream accuracy — was underpowered at this budget; switching to held-out val-BPB separated recipes cleanly. We also caught a bug: Muon ran at AdamW’s learning rate and needs ~30× higher; the fix dropped its val-BPB from ~1.99 to ~1.75 — the bug, not the optimizer. Catching these for a few hundred dollars, instead of in a multi-thousand-dollar campaign, is the staged design working. Whether val-BPB’s cheap separation predicts at scale is the open question — that’s what the transfer test measures.
The result so far
At the ~250M reference, all 18 recipes graded: ρ = 0.614, decision-accuracy = 0.74. The point estimate lands on the pre-registered GO line (ρ ≈ 0.6), but at this n the 95% confidence interval runs down to ~0.20 — below the floor the rule requires. So by our own bar this is not yet a credible signal: not a PASS, not validated, not proven. The extremes agree at both scales — but that’s the easy part; ρ is held down by a tight near-baseline cluster and one apparent discordance: the cheap gate appears to under-penalize short context (a 512-token run), though we can’t yet fully rule out noise. A larger ~1B reference is what adjudicates GO/NO-GO; this ~250M stage was the cheap pre-check, and its full results are public either way — PASS or FAIL.