# You could have invented transformers from memory

Digest AI · Research · published 2026-10-16T19:01:17Z

Canonical: https://digestai.news/story/you-could-have-invented-transformers-from-memory

## Summary

A new technical post from Hacker News explores the internal mechanics of large language models by framing linear maps, MLPs, and self-attention as different types of memory stores. The author argues that a model's ability to recall information is limited by an "interference-to-signal ratio," where stored memories interfere with each other during retrieval. The post derives heuristics for how much information each layer type can hold before this interference becomes overwhelming.

The analysis shows that a basic linear layer with 64-dimensional keys can store approximately 64 memories before recall fails. By optimizing the linear store, capacity doubles to about 128 memories. Introducing non-linearities, such as those in MLPs, allows the model to suppress low-similarity keys, increasing capacity to roughly 2,700 memories. Self-attention mechanisms offer the highest capacity, potentially storing over a million memories with the same 64-dimensional keys. This perspective provides an intuitive explanation for why attention layers are so effective at retrieving specific facts from vast datasets.

## Key points

- Linear layers store about 64 memories in 64 dimensions before interference blocks recall.
- Optimized linear stores double capacity to 128 memories by canceling key covariance.
- Self-attention mechanisms can store over a million memories using 64-dimensional keys.

## Why it matters

This research clarifies why attention mechanisms are superior to simpler layers for knowledge retrieval, offering a mathematical basis for understanding model capacity limits and architectural choices in LLMs.

## Sources

1. [You could have invented transformers from memory](https://alexwang.ai/posts/associative-memory) (alexwang.ai, 2026-10-16)

## Cite

Digest AI, "You could have invented transformers from memory", 16 October 2026, https://digestai.news/story/you-could-have-invented-transformers-from-memory

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/you-could-have-invented-transformers-from-memory.json