You could have invented transformers from memory
A new technical post from Hacker News explores the internal mechanics of large language models by framing linear maps, MLPs, and self-attention as different types of memory stores. The author argues that a model's ability to recall information is limited by an "interference-to-signal ratio," where stored memories interfere with each other during retrieval. The post derives heuristics for how…
Key points
- Linear layers store about 64 memories in 64 dimensions before interference blocks recall.
- Optimized linear stores double capacity to 128 memories by canceling key covariance.
- Self-attention mechanisms can store over a million memories using 64-dimensional keys.
The analysis shows that a basic linear layer with 64-dimensional keys can store approximately 64 memories before recall fails. By optimizing the linear store, capacity doubles to about 128 memories. Introducing non-linearities, such as those in MLPs, allows the model to suppress low-similarity keys, increasing capacity to roughly 2,700 memories. Self-attention mechanisms offer the highest capacity, potentially storing over a million memories with the same 64-dimensional keys. This perspective provides an intuitive explanation for why attention layers are so effective at retrieving specific facts from vast datasets.
Model page: Astra →
You could have invented transformers from memory
alexwang.ai · 16 October 2026
Loading the full article…
This text was published by alexwang.ai and written by Alex Wang. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Hacker News discussion · 43 pointsnews.ycombinator.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic’s Claude Mythos 5 redesigns quantum circuits better than prior research · 2 src
- Danaher and NVIDIA partner to integrate AI into scientific instruments · 3 src
- Nvidia pledges $250M in compute for UK AI agent safety research · 13 src
- LitGene transformer integrates text and ontology to improve gene representations · 1 src
- Anthropic’s Claude traces NIST ECDSA seeds to 1990s Unix timestamps · 2 src
Comments
via GitHub Discussions