# Hierarchical networks achieve up to 5× compute efficiency over tokenized Transformers

Digest AI · Research · published 2026-10-28T18:07:41Z

Canonical: https://digestai.news/story/hierarchical-networks-achieve-up-to-5-compute-efficiency-over-tokenize

## Summary

Researchers from the H‑Net project presented scaling experiments that compare traditional tokenized Transformers with their hierarchical network (H‑Net) designs. A single‑stage H‑Net, which learns tokenization inside the model, showed 3‑5× compute efficiency gains on validation loss, with larger improvements on code and out‑of‑domain tasks. A two‑stage H‑Net further reduced the resources needed, matching an open‑source baseline while using roughly ten times less compute and data.

The study used a mixed dataset composed of 70% English, 20% code, 5% math and 5% multilingual text, measured with bits‑per‑byte loss. By keeping the inner Transformer architecture identical and only swapping the tokenization component for a learned hierarchical encoder/decoder, the authors isolated the impact of dynamic tokenization. Results suggest that end‑to‑end byte‑level processing can both lower loss and cut training costs, hinting at broader applicability beyond text, such as in biology and vision.

The authors plan to release an improved H‑Net version and continue exploring its multi‑modal potential, positioning the architecture as a step toward more general learning systems that operate directly on raw data.

## Key points

- 1-stage H‑Nets provide 3‑5× compute efficiency versus tokenized Transformers by learning tokenization
- 2-stage H‑Net outperforms a matched open‑source baseline with roughly 10× less compute and data
- Experiments used a 70% English, 20% code, 5% math, 5% multilingual dataset and measured bits‑per‑byte loss

## Why it matters

Replacing hand‑crafted tokenizers with learned hierarchical chunking could make future models cheaper to train and able to ingest raw data across many domains, speeding AI development and lowering resource barriers.

## Sources

1. [Scaling Hierarchical Networks](https://cartesia.ai/blog/from-tokens-to-bytes-part-1) (cartesia.ai, 2026-10-28)

## Cite

Digest AI, "Hierarchical networks achieve up to 5× compute efficiency over tokenized Transformers", 28 October 2026, https://digestai.news/story/hierarchical-networks-achieve-up-to-5-compute-efficiency-over-tokenize

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/hierarchical-networks-achieve-up-to-5-compute-efficiency-over-tokenize.json