# Researchers extract circuits from language models using Attention routing

Digest AI · Research · published 2026-09-23T04:00:00Z

Canonical: https://digestai.news/story/researchers-extract-circuits-from-language-models-using-attention-rout

## Summary

The authors propose a method to extract circuits from language models by treating attention as a routing map from a single forward pass. They evaluate on GPT-2 Small, GPT-2 Medium, and Pythia-410M using 100 prompts per cell. Ablation of extracted edges hurts performance more than random edges. The extraction step costs one forward pass, whereas a head‑by‑head patch sweep costs about two orders of magnitude more. Code is available on GitHub.

## Key points

- Method extracts circuits using attention routing from a single forward pass
- Evaluation on GPT-2 Small, GPT-2 Medium, Pythia-410M with 100 prompts per cell
- Extraction costs one forward pass, patch sweep costs ~100x more

## Why it matters

Provides a cheap way to identify causal circuits in language models, enabling more efficient debugging and interpretability.

## Sources

1. [Attention as a Routing Graph: Live Circuit Extraction from a Single Forward Pass](https://arxiv.org/abs/2609.25285) (arXiv cs.AI, 2026-09-23, primary source)

## Cite

Digest AI, "Researchers extract circuits from language models using Attention routing", 23 September 2026, https://digestai.news/story/researchers-extract-circuits-from-language-models-using-attention-rout

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-extract-circuits-from-language-models-using-attention-rout.json