Open-source LLMS over-cooperate in prisoner's dilemma, study shows
The paper reports that open‑source large language models (LLMs) tend to cooperate far more than humans in repeated Prisoner’s Dilemma games. Researchers tested six models—TII Falcon 7B Instruct, Google Gemma 7B, Meta Llama 3.1 8B Instruct, Microsoft Phi Mini MoE Instruct, and Qwen 2.5 7B Instruct—using a standardized prompting framework.
Key points
- Open‑source llms achieved cooperation rates above 80% in shuffled‑partner setting
- Human cooperation averaged 28.5% in same setting
- Qwen‑2.5‑7B‑Instruct behaved deterministically, Phi‑Mini‑MoE‑Instruct variable
In a shuffled‑partner setting, human cooperation averaged 28.5 % over 100 rounds, while several LLMs exceeded 80 %. Qwen‑2.5‑7B‑Instruct behaved deterministically, whereas Phi‑Mini‑MoE‑Instruct showed greater variability and briefly matched human rates. The study also examined a fixed‑partner setting where both agents retained memory.
These results suggest that current open‑source LLMs are not yet reliable substitutes for human participants in behavioral simulations, highlighting the need for better memory mechanisms and more realistic decision‑making models.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic reports AI agents spend 80% of session waiting for humans, remeasurement shows about 30% · 1 src
- CoDA enables rapid codebook development for domain-specific text annotation · 1 src
- Researchers prove Prefix-Tuning can exactly steer pretrained transformers · 1 src
- Study probes if LMs understand variable roles via SAEs · 1 src
- OR-License benchmark tests LLM agents on real operations research tasks · 1 src
Comments
via GitHub Discussions