Hierarchical networks achieve up to 5× compute efficiency over tokenized Transformers
Researchers from the H‑Net project presented scaling experiments that compare traditional tokenized Transformers with their hierarchical network (H‑Net) designs. A single‑stage H‑Net, which learns tokenization inside the model, showed 3‑5× compute efficiency gains on validation loss, with larger improvements on code and out‑of‑domain tasks. A two‑stage H‑Net further reduced the resources needed,…
Key points
- 1-stage H‑Nets provide 3‑5× compute efficiency versus tokenized Transformers by learning tokenization
- 2-stage H‑Net outperforms a matched open‑source baseline with roughly 10× less compute and data
- Experiments used a 70% English, 20% code, 5% math, 5% multilingual dataset and measured bits‑per‑byte loss
The study used a mixed dataset composed of 70% English, 20% code, 5% math and 5% multilingual text, measured with bits‑per‑byte loss. By keeping the inner Transformer architecture identical and only swapping the tokenization component for a learned hierarchical encoder/decoder, the authors isolated the impact of dynamic tokenization. Results suggest that end‑to‑end byte‑level processing can both lower loss and cut training costs, hinting at broader applicability beyond text, such as in biology and vision.
The authors plan to release an improved H‑Net version and continue exploring its multi‑modal potential, positioning the architecture as a step toward more general learning systems that operate directly on raw data.
Scaling Hierarchical Networks
cartesia.ai · 28 October 2026
Loading the full article…
This text was published by cartesia.ai and written by Cartesia Research. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Hacker News discussion · 45 pointsnews.ycombinator.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic says unreleased model built working exploits for Chrome, Safari, Windows, Linux · 2 src
- OpenAI and Anthropic claim their AI models refuted the Exponential Time Hypothesis · 13 src
- Cprisp framework predicts circRNA-RBP interactions for neurodegeneration research · 1 src
- TinyLeague updates benchmarks: drops AA Index, adds KV efficiency metrics · 1 src
- Artificial Analysis releases Intelligence Index v5 with private datasets · 1 src
Comments
via GitHub Discussions