{"version":1,"type":"story","url":"https://digestai.news/story/you-could-have-invented-transformers-from-memory","json":"https://digestai.news/story/you-could-have-invented-transformers-from-memory.json","markdown":"https://digestai.news/story/you-could-have-invented-transformers-from-memory.md","slug":"you-could-have-invented-transformers-from-memory","headline":"You could have invented transformers from memory","summary":"A new technical post from Hacker News explores the internal mechanics of large language models by framing linear maps, MLPs, and self-attention as different types of memory stores. The author argues that a model's ability to recall information is limited by an \"interference-to-signal ratio,\" where stored memories interfere with each other during retrieval. The post derives heuristics for how much information each layer type can hold before this interference becomes overwhelming.\n\nThe analysis shows that a basic linear layer with 64-dimensional keys can store approximately 64 memories before recall fails. By optimizing the linear store, capacity doubles to about 128 memories. Introducing non-linearities, such as those in MLPs, allows the model to suppress low-similarity keys, increasing capacity to roughly 2,700 memories. Self-attention mechanisms offer the highest capacity, potentially storing over a million memories with the same 64-dimensional keys. This perspective provides an intuitive explanation for why attention layers are so effective at retrieving specific facts from vast datasets.","keyPoints":["Linear layers store about 64 memories in 64 dimensions before interference blocks recall.","Optimized linear stores double capacity to 128 memories by canceling key covariance.","Self-attention mechanisms can store over a million memories using 64-dimensional keys."],"whyItMatters":"This research clarifies why attention mechanisms are superior to simpler layers for knowledge retrieval, offering a mathematical basis for understanding model capacity limits and architectural choices in LLMs.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":[],"models":["Astra"],"people":[]},"firstPublishedAt":"2026-10-16T19:01:17Z","updatedAt":"2026-10-16T19:01:17Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"alexwang.ai","title":"You could have invented transformers from memory","url":"https://alexwang.ai/posts/associative-memory","publishedAt":"2026-10-16T19:01:17Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[{"site":"Hacker News","url":"https://news.ycombinator.com/item?id=50122878","points":43}],"thread":null,"cite":{"text":"Digest AI, \"You could have invented transformers from memory\", 16 October 2026, https://digestai.news/story/you-could-have-invented-transformers-from-memory","publisher":"Digest AI","title":"You could have invented transformers from memory","datePublished":"2026-10-16T19:01:17Z","url":"https://digestai.news/story/you-could-have-invented-transformers-from-memory"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}