Researchers prune 729 attention heads to cut GPT-2-XL’s memorisation by 91.5%
A new paper from arXiv explores how removing redundant attention heads in large language models can reduce verbatim memorisation of training data. The authors found that zeroing out 729 heads in GPT-2-XL (1.5B parameters) slashed memorised-sequence reproduction by 91.5% while improving language modelling performance on WikiText-103 by 0.92 perplexity points. This approach avoids retraining or…
Key points
- Pruning 729 redundant attention heads in GPT-2-XL cuts memorised-sequence recall by 91.5% during inference
- Same pruning improves WikiText-103 perplexity by 0.92 points, per the paper’s benchmarks
- No retraining or data modification required—change happens only at inference time
The work builds on prior findings that many attention heads are functionally redundant and can be pruned without harming core performance. By leveraging this redundancy, the researchers propose a method to mitigate privacy and IP risks tied to memorisation, though the technique remains theoretical and untested in production models.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- SemiAnalysis releases GDPx benchmark with Claude Opus 5.5 leading at 12% · 3 src
- Researchers release Qistas protocol for testing LLM fairness across cultures · 1 src
- Researchers compare speculative drafters and continuous diffusion language models for text generation · 1 src
- Researchers propose Li²SAO to improve asynchronous RL training for LLMs · 1 src
- Researchers study how LLMs assign emotional value to outcomes · 1 src
Comments
via GitHub Discussions