Researchers find transformer prompt influence decays as power law
The paper, posted on arXiv, examines how much of a transformer model’s prompt remains after generating tokens. The authors show that the distance between hidden states of two different prompts decays approximately as a power law $t^{-alpha}$, with $alpha=1$ for a fixed prompt’s share of uniform attention. Trained models forget fast ($alphaapprox0.55$–$0.7$), untrained slowly…
1 source primary source
Key points
- Prompt influence decays as $t^{-alpha}$ with $alpha\approx0.55$–$0.7$ for trained models
- Upper bound $O\big(P(\log t)^{L-1}/t\big)$ holds for finite‑depth pre‑LN softmax transformers
- Conditional no‑hair theorem: prompt influence vanishes as $t$ grows, leaving only text to carry prompt properties
Read the original at arXiv cs.AI · by Shuyang Xiang primary sourceOpen source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic reports AI agents spend 80% of session waiting for humans, remeasurement shows about 30% · 1 src
- CoDA enables rapid codebook development for domain-specific text annotation · 1 src
- Researchers prove Prefix-Tuning can exactly steer pretrained transformers · 1 src
- Study probes if LMs understand variable roles via SAEs · 1 src
- OR-License benchmark tests LLM agents on real operations research tasks · 1 src
Comments
via GitHub Discussions