PsyAgentBench tests if LLMs truly mimic human psychological biases

Results show that human-like effects in LLMs arise through different mechanisms rather than a single susceptibility. For example, Asch conformity effects jumped from 0 percent in blind trials to 83.3 percent in named trials for the gpt-oss-120B model. Anchoring effects were absent when grounded in facts but strong with invented quantities, suggesting rational reliance on available signals. Sunk cost effects were largely absent, while minimal-group allocation was dominated by safety-mediated refusals.

The authors argue that simple scalar bias scores obscure these complex structures. They identify three ways psychology paradigms fail to port to LLMs: persona dominance, population collapse, and safety selection. A one-sentence persona change could eliminate, dampen, or reverse these effects, challenging the idea of a unified response bias in large language models.