Zyphra releases ZHA, a sparse attention method for frozen models
Zyphra has released Zyphra Hierarchical Attention (ZHA), a sparse attention method designed to reduce the computational cost of long-context processing. Unlike existing sparse methods that still scan the entire context to find relevant tokens, ZHA uses a tree-structured indexer to search for keys hierarchically. This approach allows the model to keep the full KV cache while only reading the most…
Key points
- ZHA adds 81,920 trainable parameters to frozen models like Qwen3.6-35B-A3B.
- Deep-tree ZHA scores 63.4% on LongBench v2, beating dense attention's 58.0%.
- Speedups up to 417x at 1B tokens are modeled projections, not measured results.
The method is lightweight, adding only 81,920 trainable parameters to a frozen pretrained model, such as Qwen3.6-35B-A3B. Zyphra reports that ZHA can be distilled into these models using approximately 4B tokens of training data. In benchmarks, the deep-tree variant of ZHA achieved 63.4% on LongBench v2, outperforming dense attention's 58.0% while using only about 1.1% of keys. It also maintained high accuracy on RULER-13, staying within a few points of dense attention across reasoning and coding tasks.
However, performance varies by base model. On Nemotron-3.5-Lightning-30B-A3B, the deep-tree configuration dropped 25.3 points below dense attention on RULER-13. Speed improvements are currently theoretical; Zyphra has not released optimized kernels, and reported speedups of 373 to 417x at 1B tokens are based on modeled H100 BF16 roofline projections rather than measured benchmarks.
Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention
MarkTechPost · 3 November 2026
Loading the full article…
This text was published by MarkTechPost and written by Asif Razzaq. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- MIT researchers report GPT-6 Astra natively reads and writes neural network weights, raising Qwen3.5-0.8B MATH-500 from 7.9% to 48.1% by editing 0.016% of weigh · 3 src
- University of Washington study finds AI models penalize Chinese and Nigerian accents · 1 src
- 10x Genomics integrates Anthropic’s Claude into Sentira Single Cell for faster analysis · 1 src
- Researchers demonstrate pretraining poisoning backdoors in Qwen3 agents · 1 src
- Researchers introduce PathLUPI to improve molecular prediction from histology images · 1 src
Comments
via GitHub Discussions