Researchers introduce XiangqiBench to evaluate closed-loop performance of LLM agents in Chinese chess

The authors recorded 8,568 multi‑turn trajectories from 12 frontier LLMs under two observation protocols via an interactive REPL that separates real moves, state queries, and forward simulation. Their analysis reveals three gaps: the Conversion Gap (models play the stored reference first move in 26.1 % of sighted trials but only 13.9 % of those end in a win), the Consistency Gap (the leading model reaches 38.7 % pass@3 yet only 5.9 % pass³, winning all three trials on 7 of the 46 positions it ever wins), and the Simulation Gap (32.3 % of accepted simulation calls stop on an illegal move, and in 49.3 % of comparable cases the real defender replies differently from the simulated line). The authors argue that agent evaluations should report closed‑loop outcomes and reliability alongside coverage.