# Author pretrains language model End-to-End in Rust for $164

Digest AI · Research · published 2026-09-23T04:00:00Z

Canonical: https://digestai.news/story/author-pretrains-language-model-end-to-end-in-rust-for-164

## Summary

During the run the author catalogued failures in the two leading Rust training frameworks, Candle and Burn. Five Candle defects were found, including fused kernels that silently produced no gradient, while three Burn defects included a backward pass that achieved only about 3 % of theoretical GPU throughput and a kernel‑fusion path that segfaulted at multi‑billion‑parameter scale. A gradient‑flow arbiter test was introduced to catch such silent failures.

The resulting model achieved a per‑token negative log‑likelihood of 0.93 versus 12.60 for a random‑initialized twin, but performed at chance on English commonsense multiple‑choice, as expected from the Bangla‑weighted training budget. The author also uncovered a tokenizer‑fertility trap that collapsed Bangla tokens to 1.4 characters per token, later corrected to 4.1. After the experiment the author moved training to PyTorch and retained Rust for on‑device serving, noting Rust is not yet competitive for training but may be suitable for serving.

## Key points

- identified five Candle and three Burn training defects, including silent gradient loss and low throughput
- gradient‑flow arbiter test ensures every parameter receives finite, nonzero gradient

## Why it matters

The report highlights practical limitations of Rust‑based training frameworks and offers a verification method that could improve reliability for future Rust‑based AI research.

## Sources

1. [Training a Language Model End-to-End in Rust: An Experience Report](https://arxiv.org/abs/2609.25008) (arXiv cs.CL, 2026-09-23, primary source)

## Cite

Digest AI, "Author pretrains language model End-to-End in Rust for $164", 23 September 2026, https://digestai.news/story/author-pretrains-language-model-end-to-end-in-rust-for-164

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/author-pretrains-language-model-end-to-end-in-rust-for-164.json