# Researchers prune 729 attention heads to cut GPT-2-XL’s memorisation by 91.5%

Digest AI · Research · published 2026-11-02T05:00:00Z

Canonical: https://digestai.news/story/researchers-prune-729-attention-heads-to-cut-gpt-2-xls-memorisation-by

## Summary

A new paper from arXiv explores how removing redundant attention heads in large language models can reduce verbatim memorisation of training data. The authors found that zeroing out 729 heads in GPT-2-XL (1.5B parameters) slashed memorised-sequence reproduction by 91.5% while improving language modelling performance on WikiText-103 by 0.92 perplexity points. This approach avoids retraining or altering training data, instead targeting inference-time behaviour.

The work builds on prior findings that many attention heads are functionally redundant and can be pruned without harming core performance. By leveraging this redundancy, the researchers propose a method to mitigate privacy and IP risks tied to memorisation, though the technique remains theoretical and untested in production models.

## Key points

- Pruning 729 redundant attention heads in GPT-2-XL cuts memorised-sequence recall by 91.5% during inference
- Same pruning improves WikiText-103 perplexity by 0.92 points, per the paper’s benchmarks
- No retraining or data modification required—change happens only at inference time

## Why it matters

If scalable, this method could let model operators reduce privacy risks without rebuilding models from scratch.

## Sources

1. [Fewer Heads, Fewer Memories: Pruning Redundant Attention Heads Suppresses Verbatim Memorisation](https://arxiv.org/abs/2610.40974) (arXiv cs.CL, 2026-11-02, primary source)

## Cite

Digest AI, "Researchers prune 729 attention heads to cut GPT-2-XL’s memorisation by 91.5%", 2 November 2026, https://digestai.news/story/researchers-prune-729-attention-heads-to-cut-gpt-2-xls-memorisation-by

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-prune-729-attention-heads-to-cut-gpt-2-xls-memorisation-by.json