Arxiv paper introduces Boa Compressor for Linear-Time context compression
The paper proposes Boa Compressor, a plug‑and‑play online context compressor built on a Mamba state‑space model. It encodes arbitrary‑length context in a single linear‑time pass and produces a compressed representation that a frozen LLM can condition on.
Key points
- Boa Compressor encodes arbitrary‑length context in linear time using a Mamba state‑space model.
- It matches or outperforms offline baselines while being ten times faster than Transformer‑based compressors.
The compressor is trained by distillation, matching the next‑token logits of a teacher LLM that sees the full context with a student LLM that sees the compressed representation. At equal compression ratios, Boa matches or surpasses state‑of‑the‑art offline baselines while being an order of magnitude faster than Transformer‑based implementations, reducing end‑to‑end latency.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- OpenAI model allegedly disproves Exponential Time Hypothesis · 2 src
- Anthropic adds self-assessment layer to Claude safety monitoring · 2 src
- OpenAI claims AI models prove 1991 math conjecture in all dimensions · 2 src
- IEEE launches AI Research Suite for Xplore Digital Library · 1 src
- OpenAI releases ChatGPT work usage data for 165 activities · 1 src
Comments
via GitHub Discussions