Research · Updated 15 Oct, 04:17 pm IST
Google reproduces Olmo 3 7B pre-training on Cloud TPUs using MaxText
Why it matters for readers: Gives evidence that publicly documented models can be faithfully reproduced across frameworks and hardware.
- Google reproduced Ai2's Olmo 3 7B stage-1 pre-training and stage-2 mid-training anneal on Cloud TPUs with MaxText.1
- They converted the PyTorch Olmo 3 architecture to JAX/MaxText and achieved a logit-parity KL ≈ 1.5e-3 versus the HuggingFace reference.1
- At full 8,192-token context in bfloat16 the converted model agreed with the reference on the top-1 token 98.75% of the time.1
- Held-out evaluations during the reproduction process revealed a data-loader bug in MaxText that would have misleadingly suggested better performance.1
Get a brief like this every morning
Uzha reads hundreds of sources and gives you the stories that matter for your work, with every source linked. Free.
Get started