# OR-License benchmark tests LLM agents on real operations research tasks

Digest AI · Research · published 2026-10-20T04:00:00Z

Canonical: https://digestai.news/story/or-license-benchmark-tests-llm-agents-on-real-operations-research-task

## Summary

Researchers introduced OR-License, a new benchmark designed to evaluate large language model agents on real-world operations research (OR) problems. Unlike existing benchmarks that use simplified or predefined tasks, OR-License contains 31 instances derived from 10 industrial scenarios based on paid OR engagements totaling USD 3.68 million. The benchmark preserves original client requirements, data, and deliverable formats after de-identification to test if LLMs can translate raw business materials into professional solutions without prior expert reformulation.

To assess how agents handle information gaps, the team developed ORSim, a controlled client simulator that discloses business facts without providing OR expertise. The evaluation uses Deployment-Adjusted Quality (DAQ), which averages expert-relative quality and assigns zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several models exceeded the expert reference in conditional objective quality, but none matched the expert’s overall DAQ. The strongest configuration, Claude Code, achieved 88.39% strict feasibility and a DAQ of 0.87 compared to the expert reference of 1.

Trajectory analysis revealed that failures often stem from formulation and implementation errors. Task-equalized research-chain completion rates ranged from 24% to 45% across models. The study highlights the need for agents to better resolve information gaps and maintain faithful problem formulation to achieve reliable long-horizon performance in complex industrial settings.

## Key points

- OR-License benchmark uses 31 instances from USD 3.68 million in real OR engagements.
- No LLM matched expert DAQ; best Claude Code config scored 0.87 versus 1.0.
- Research-chain completion rates ranged from 24% to 45% across 11 frontier LLMs.

## Why it matters

This benchmark provides a rigorous measure of LLM capability in complex, real-world industrial optimization, highlighting significant gaps between current agent performance and human expert reliability in high-stakes operations research.

## Sources

1. [OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners](https://arxiv.org/abs/2610.24087) (arXiv cs.CL, 2026-10-20, primary source)

## Cite

Digest AI, "OR-License benchmark tests LLM agents on real operations research tasks", 20 October 2026, https://digestai.news/story/or-license-benchmark-tests-llm-agents-on-real-operations-research-task

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/or-license-benchmark-tests-llm-agents-on-real-operations-research-task.json