OR-License benchmark tests LLM agents on real operations research tasks
Researchers introduced OR-License, a new benchmark designed to evaluate large language model agents on real-world operations research (OR) problems. Unlike existing benchmarks that use simplified or predefined tasks, OR-License contains 31 instances derived from 10 industrial scenarios based on paid OR engagements totaling USD 3.68 million. The benchmark preserves original client requirements,…
Key points
- OR-License benchmark uses 31 instances from USD 3.68 million in real OR engagements.
- No LLM matched expert DAQ; best Claude Code config scored 0.87 versus 1.0.
- Research-chain completion rates ranged from 24% to 45% across 11 frontier LLMs.
To assess how agents handle information gaps, the team developed ORSim, a controlled client simulator that discloses business facts without providing OR expertise. The evaluation uses Deployment-Adjusted Quality (DAQ), which averages expert-relative quality and assigns zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several models exceeded the expert reference in conditional objective quality, but none matched the expert’s overall DAQ. The strongest configuration, Claude Code, achieved 88.39% strict feasibility and a DAQ of 0.87 compared to the expert reference of 1.
Trajectory analysis revealed that failures often stem from formulation and implementation errors. Task-equalized research-chain completion rates ranged from 24% to 45% across models. The study highlights the need for agents to better resolve information gaps and maintain faithful problem formulation to achieve reliable long-horizon performance in complex industrial settings.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- CoDA enables rapid codebook development for domain-specific text annotation · 1 src
- Researchers prove Prefix-Tuning can exactly steer pretrained transformers · 1 src
- Study probes if LMs understand variable roles via SAEs · 1 src
- Study finds most LLM agent decisions on prompt injections are set at read time · 1 src
- Researchers test open-weight Hunyuan-MT-7B for stylistic translation · 1 src
Comments
via GitHub Discussions