Ralph-1, and a number for what “real improvement” means
Before a network can crown the best training recipe, it needs to know what counts as better at all. So the first real run wasn’t about the model — it was about measuring the noise.
The baseline
The canonical recipe trained Ralph-1 — 253,872,128 parameters, 262,144,000 FineWeb-Edu tokens (~262M, from a 1B-token tokenized corpus) — to a final validation loss of 3.8163 in bf16, in 69 minutes on a single H100. The model is the byproduct. What mattered was the calibration that came with it.
The noise floor
We trained the same 125M configuration across 10 seeds, 500 steps each, and measured the spread in held-out val-BPB. The standard deviation was σ = 0.0064. That sets the bar: a proposed recipe change has to beat the sitting king by more than 2σ = 0.013 val-BPB to count as signal rather than seed luck. That threshold isn’t a vibe — it lives in the validator code as a literal constant, and it’s what every king change since has had to clear.
Why measure noise before anything else
You can’t call a recipe change an improvement until you know how much the score moves on seed alone. So before scoring a single proposed change, we pinned the seed-to-seed variation of the unchanged recipe. That number — 2σ = 0.013 val-BPB — is what turns ‘decisively beats the king’ from a vibe into a literal threshold in the validator code. Every king change since has had to clear it, and every evaluated change, win or loss, is published with its multi-seed variance. Honest limit: this is a single-box H100 measurement on FineWeb-Edu, and Ralph-1 is a deliberately short baseline — its job is to anchor the lineage cheaply, not to be a strong model.