Researchers propose Joint Subspace Sparse Attention for faster long-context decoding
A new paper on arXiv introduces Joint Subspace Sparse Attention (JSSA), a method to speed up long-context decoding in large language models. The technique reduces KV-cache traffic—a bottleneck for long prompts—by selecting tokens at a fine granularity while preserving accuracy. Unlike prior methods, JSSA avoids latency trade-offs by learning a shared low-dimensional subspace for query-key…
Key points
- JSSA learns a shared query-key subspace to improve token selection accuracy without retraining models
- Method achieves **6.43× faster decoding** than full-KV attention at **128K context** on NVIDIA B200 GPUs
- Works across GQA, hybrid sliding-window, and MLA architectures with **96–99% throughput retention** at 700B scale
On an NVIDIA B200 GPU, JSSA achieves 6.43× faster decoding than full-KV attention at 128K context length while maintaining reasoning accuracy comparable to model-native DSA. It also scales efficiently to 700B-scale models, retaining 96–99% of full-KV throughput even at high context lengths. The authors claim the method avoids generation-length inflation and architecture-dependent failures seen in competing sparse attention approaches.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic reports small AI models may self-replicate on IoT devices within months · 25 src
- MIT professor Eric So warns of AI cognitive atrophy · 1 src
- Datalab releases OmniParseBench, an open OCR benchmark with 16,288 tests · 1 src
- Meta engineer claims Anthropic's Scion copies Mononoke design · 1 src
- Colossal introduces GenoMorph AI to target dodo gene-editing sites · 1 src
Comments
via GitHub Discussions