{"version":1,"type":"story","url":"https://digestai.news/story/google-releases-85-task-engineering-benchmark-and-agent-competition-wi","json":"https://digestai.news/story/google-releases-85-task-engineering-benchmark-and-agent-competition-wi.json","markdown":"https://digestai.news/story/google-releases-85-task-engineering-benchmark-and-agent-competition-wi.md","slug":"google-releases-85-task-engineering-benchmark-and-agent-competition-wi","headline":"Google releases 85-task engineering benchmark and agent competition with Mercor","summary":"Google AI & Infrastructure opened a new benchmark and competition on November 3, 2026, called Physical-Engineering’s Last Exam (PLE). Developed with Mercor, PLE tests agents on 85 real-world engineering tasks—30 mechanical, 27 electrical, 18 civil, and 10 structural—requiring deliverables like memos, drawings, and schedules that meet professional standards and codes. The benchmark includes 5,170 files across formats, including CAD models and BIM files, and was designed by practicing engineers to simulate professional workflows, not textbook problems. Grading emphasizes completed, compliant work, with hard safety and feasibility gates that can invalidate entire designs if violated, even if most requirements are met.\n\nGoogle evaluated 14 models using seven agent harnesses, finding all scored below 50% on the private test set. Claude Opus 5.5 in Claude Code led with 48.3% on its native harness. The paper notes a gap between satisfying individual constraints (91% for the top model) and full deliverable compliance (41.3%). The competition, which launched alongside the benchmark, requires entrants to submit an agent harness, model choice, and artifacts for public tasks. Grading includes reruns on hidden tasks to determine leaderboard scores. The Apache-2.0-licensed repository provides starter code and supports 13 Google-sponsored models, commercial APIs, or self-hosted models. The focus is on end-to-end compliance, not just correctness, exposing a key challenge in real-world agent deployment.","keyPoints":["PLE benchmark includes 85 engineering tasks across mechanical, electrical, civil, and structural disciplines, with 5,170 files total","Google reports all 14 tested models scored below 50% on hidden test set, with Claude Opus 5.5 leading at 48.3%","Competition evaluates full agent systems, including harnesses, with grading requiring deliverables to pass hard safety and feasibility checks"],"whyItMatters":"PLE forces agents to handle real-world engineering complexity, not just isolated tasks, revealing gaps in compliance and deliverable quality. The competition could accelerate progress in agent systems for professional workflows, but the 50-point model spread highlights how far agents are from reliable engineering support.","category":{"slug":"agents","name":"Agents & Tools","url":"https://digestai.news/category/agents"},"entities":{"companies":["Google AI & Infrastructure","Mercor"],"models":["Claude Opus 5.5"],"people":[]},"firstPublishedAt":"2026-11-03T17:25:03Z","updatedAt":"2026-11-03T17:25:03Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"Unite.AI","title":"Google Opens 85-Task Engineering Benchmark and Agent Competition","url":"https://unite.ai/google-opens-85-task-engineering-benchmark-and-agent-competition","publishedAt":"2026-11-03T17:25:03Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Google releases 85-task engineering benchmark and agent competition with Mercor\", 3 November 2026, https://digestai.news/story/google-releases-85-task-engineering-benchmark-and-agent-competition-wi","publisher":"Digest AI","title":"Google releases 85-task engineering benchmark and agent competition with Mercor","datePublished":"2026-11-03T17:25:03Z","url":"https://digestai.news/story/google-releases-85-task-engineering-benchmark-and-agent-competition-wi"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}