Researchers find instruction exposure unlocks rhetorical patterns in LLMs
A new study on arXiv explores how language models absorb rhetorical strategies during pretraining, not just facts. The authors extracted an ‘Analogy Vector’ from residual streams to measure how much these models can be steered toward analogy-based reasoning. Testing seven instruction-tuned models—including Gemma-2, Llama-3.1, and Qwen-2.5—they found a 22% improvement in analogy quality for…
Key points
- Analogy Vector extracted from residual streams improves analogy reasoning by 22% for Gemma-2-9B-IT
- Qwen-2.5 base models show higher cosine similarity (0.94–0.97) than Gemma-2 (0.37–0.66) due to instruction exposure
- Study challenges ‘base vs. instruct’ framing, arguing instruction exposure during pretraining unlocks rhetorical patterns
The research reveals a gradient of behavioral accessibility: models with instruction data woven into pretraining (like Qwen-2.5) show higher cosine similarity scores (0.94–0.97) for the analogy vector, compared to models with strict pretraining/instruction separation (Gemma-2, scoring 0.37–0.66). This challenges the binary ‘base vs. instruct’ framing, suggesting that the degree of instruction exposure during training determines how easily rhetorical patterns can be activated at inference time.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- AI startups aim to automate formal software verification with $365M funding · 2 src
- Scoring Explanations Without Reading Them: Structural Shortcuts in Automated-Interpretability · 1 src
- Researchers introduce FilingLoop for SEC disclosure review · 1 src
- Researchers propose Verification Learning Theory to model AI-assisted judgment checks · 1 src
- Researchers release LibStructBench to test AI agents’ sequencing protocol accuracy · 1 src
Comments
via GitHub Discussions