Author pretrains language model End-to-End in Rust for $164
During the run the author catalogued failures in the two leading Rust training frameworks, Candle and Burn. Five Candle defects were found, including fused kernels that silently produced no gradient, while three Burn defects included a backward pass that achieved only about 3 % of theoretical GPU throughput and a kernel‑fusion path that segfaulted at multi‑billion‑parameter scale. A…
Key points
- identified five Candle and three Burn training defects, including silent gradient loss and low throughput
- gradient‑flow arbiter test ensures every parameter receives finite, nonzero gradient
The resulting model achieved a per‑token negative log‑likelihood of 0.93 versus 12.60 for a random‑initialized twin, but performed at chance on English commonsense multiple‑choice, as expected from the Bangla‑weighted training budget. The author also uncovered a tokenizer‑fertility trap that collapsed Bangla tokens to 1.4 characters per token, later corrected to 4.1. After the experiment the author moved training to PyTorch and retained Rust for on‑device serving, noting Rust is not yet competitive for training but may be suitable for serving.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Semantic Routing Calibration mitigates LLM over-refusal · 1 src
- AIBuildAI-2.5 ranks first on MLE-Bench with 73.3% medal rate · 1 src
- Researchers propose latent equivalence learning for enterprise data agents, score 94.67% on benchmark · 1 src
- Researchers extract circuits from language models using Attention routing · 1 src
- ReAdapt improves warm‑introduction and reaction selection accuracy for Gemini‑3‑Flash · 1 src
Comments
via GitHub Discussions