{"version":1,"type":"story","url":"https://digestai.news/story/or-license-benchmark-tests-llm-agents-on-real-operations-research-task","json":"https://digestai.news/story/or-license-benchmark-tests-llm-agents-on-real-operations-research-task.json","markdown":"https://digestai.news/story/or-license-benchmark-tests-llm-agents-on-real-operations-research-task.md","slug":"or-license-benchmark-tests-llm-agents-on-real-operations-research-task","headline":"OR-License benchmark tests LLM agents on real operations research tasks","summary":"Researchers introduced OR-License, a new benchmark designed to evaluate large language model agents on real-world operations research (OR) problems. Unlike existing benchmarks that use simplified or predefined tasks, OR-License contains 31 instances derived from 10 industrial scenarios based on paid OR engagements totaling USD 3.68 million. The benchmark preserves original client requirements, data, and deliverable formats after de-identification to test if LLMs can translate raw business materials into professional solutions without prior expert reformulation.\n\nTo assess how agents handle information gaps, the team developed ORSim, a controlled client simulator that discloses business facts without providing OR expertise. The evaluation uses Deployment-Adjusted Quality (DAQ), which averages expert-relative quality and assigns zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several models exceeded the expert reference in conditional objective quality, but none matched the expert’s overall DAQ. The strongest configuration, Claude Code, achieved 88.39% strict feasibility and a DAQ of 0.87 compared to the expert reference of 1.\n\nTrajectory analysis revealed that failures often stem from formulation and implementation errors. Task-equalized research-chain completion rates ranged from 24% to 45% across models. The study highlights the need for agents to better resolve information gaps and maintain faithful problem formulation to achieve reliable long-horizon performance in complex industrial settings.","keyPoints":["OR-License benchmark uses 31 instances from USD 3.68 million in real OR engagements.","No LLM matched expert DAQ; best Claude Code config scored 0.87 versus 1.0.","Research-chain completion rates ranged from 24% to 45% across 11 frontier LLMs."],"whyItMatters":"This benchmark provides a rigorous measure of LLM capability in complex, real-world industrial optimization, highlighting significant gaps between current agent performance and human expert reliability in high-stakes operations research.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":[],"models":["Claude Code"],"people":[]},"firstPublishedAt":"2026-10-20T04:00:00Z","updatedAt":"2026-10-20T04:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"arXiv cs.CL","title":"OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners","url":"https://arxiv.org/abs/2610.24087","publishedAt":"2026-10-20T04:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"OR-License benchmark tests LLM agents on real operations research tasks\", 20 October 2026, https://digestai.news/story/or-license-benchmark-tests-llm-agents-on-real-operations-research-task","publisher":"Digest AI","title":"OR-License benchmark tests LLM agents on real operations research tasks","datePublished":"2026-10-20T04:00:00Z","url":"https://digestai.news/story/or-license-benchmark-tests-llm-agents-on-real-operations-research-task"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}