arXiv study finds language models encode chemistry knowledge before output
A new paper on arXiv examines how five open-source language models process chemical functional groups. Researchers tested models on 92 molecules and 20 functional groups, finding that even without chemistry training, they organize functional groups by chemical family and encode chemical differences in consistent ways. For example, models place thiols (R–SH) near alcohols (R–OH) in their internal…
Key points
- Five open-source models organize chemical functional groups by family in internal representations, even without chemistry training
- Base model correctly identifies functional groups 58.5% of the time, but errors stem from silence, not wrong answers
- Chemistry fine-tuning reorganizes model focus from prompt wording to functional-group structure
The base model correctly named functional groups only 58.5% of the time, but its errors were mostly failures to respond rather than incorrect answers. Fine-tuning for chemistry shifted the models’ focus from prompt wording to functional-group structure. The study warns that relying solely on model outputs to evaluate chemical knowledge may underestimate the underlying information they encode.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic releases FlashProve open-source system for provable AI inference · 8 src
- Gemini and Claude debate AI personality, ending in a three-way tie · 1 src
- Tencent releases WeVisOne-VL, a 2B/4B open vision model for detection, OCR and matting · 1 src
- Lama-dt outperforms baselines on meta-world and mujoco · 1 src
- Researchers release Dory1.0-4B, a 4B-parameter looped model with 256K-token context · 1 src
Comments
via GitHub Discussions