Researchers find self‑rewarded moral argumentation makes language models decisive but pliable
A new arXiv paper investigates how rewarding language models for producing high‑quality moral arguments affects their judgments. The authors train a reinforcement‑learning policy, starting from a state‑of‑the‑art model, to judge its own generated arguments in a self‑judged tournament that uses masked thinking. Authentic dilemmas are extracted from classic novels and verified with a logic solver…
Key points
- Self‑rewarded RL policy judges its own arguments using masked thinking.
- Policy matches human annotators on argument quality and becomes more decisive on MoralChoice, ELEPHANT, MORABLES.
- Consistency improves under reordering, but sycophancy and validity remain challenges.
The study reports that the policy’s internal judge rates argument quality similarly to human annotators, and the model adopts the linguistic register it rewards. When evaluated on external benchmarks MoralChoice, ELEPHANT, and MORABLES, the policy becomes more decisive and invariant to the order of options, though it still exhibits sycophancy and reframing issues. Two ablation experiments with synthetic dilemmas and unmasked thinking replicate the ordering gains but show weaker register shifts and higher sycophancy. The authors conclude that consistency under reordering can be optimized without improving the underlying moral validity of the model’s judgments.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- MIT historian says early AI engineers added randomness to mimic human creativity · 1 src
- TauIndianBankBench evaluates procedure compliance in indian banking assistants · 1 src
- Lighthouse chatbot to our knowledge launched for Civil Rights history research · 1 src
- Researchers introduce MemPABench to evaluate personal agent memory · 1 src
- BELT reduces LLM reasoning tokens by 41% on Qwen3-1.7B · 1 src
Comments
via GitHub Discussions