ContextAdapt evaluates LLMs on value alignment across medicine, law, finance, and national security
The paper introduces ContextAdapt, an evaluation framework to test whether large language models appropriately apply values like honesty, autonomy, and confidentiality across professional domains while remaining consistent when context does not change the relevant norm. Researchers evaluated 12 LLMs on recommended actions and justifications using scenarios based on primary-source professional…
1 source primary source
Key points
- ContextAdapt evaluates LLMs on honesty, autonomy, and confidentiality across medicine, law, finance, and national security
- Models achieved 95.6% mean appropriateness but correct justification ranged from 25.6% to 76.9%
- Varying stakes caused localized failures where models changed responses despite unchanged obligations
Read the original at arXiv cs.CL · by Olivia Macmillan-Scott, Mirco Musolesi primary sourceOpen source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model · 1 src
- Korean legal study finds KLUE-BERT outperforms GPT models in sexual offense text classification · 1 src
- Researchers question human-derived bias measures for LLM evaluation · 1 src
- arXiv study finds reading LLM judges from first token overstates position bias · 1 src
- Study compares On-Device NER models for speed, cost and accuracy · 1 src
Comments
via GitHub Discussions