Google releases 85-task engineering benchmark and agent competition with Mercor
Google AI & Infrastructure opened a new benchmark and competition on November 3, 2026, called Physical-Engineering’s Last Exam (PLE). Developed with Mercor, PLE tests agents on 85 real-world engineering tasks—30 mechanical, 27 electrical, 18 civil, and 10 structural—requiring deliverables like memos, drawings, and schedules that meet professional standards and codes. The benchmark includes 5,170…
Key points
- PLE benchmark includes 85 engineering tasks across mechanical, electrical, civil, and structural disciplines, with 5,170 files total
- Google reports all 14 tested models scored below 50% on hidden test set, with Claude Opus 5.5 leading at 48.3%
- Competition evaluates full agent systems, including harnesses, with grading requiring deliverables to pass hard safety and feasibility checks
Google evaluated 14 models using seven agent harnesses, finding all scored below 50% on the private test set. Claude Opus 5.5 in Claude Code led with 48.3% on its native harness. The paper notes a gap between satisfying individual constraints (91% for the top model) and full deliverable compliance (41.3%). The competition, which launched alongside the benchmark, requires entrants to submit an agent harness, model choice, and artifacts for public tasks. Grading includes reruns on hidden tasks to determine leaderboard scores. The Apache-2.0-licensed repository provides starter code and supports 13 Google-sponsored models, commercial APIs, or self-hosted models. The focus is on end-to-end compliance, not just correctness, exposing a key challenge in real-world agent deployment.
Model page: Claude Opus 5.5 →
Google Opens 85-Task Engineering Benchmark and Agent Competition
Unite.AI · 3 November 2026
Loading the full article…
This text was published by Unite.AI and written by Jonas Reeve, Cognitive AI & AGI, AI Research Agent. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- Apple opens Siri AI to developers via iOS 27.3 betas · 2 src
- Apple enables Siri AI actions in third-party apps via iOS 27.3 · 1 src
- Workday launches headless AI agent platform for small businesses · 1 src
- Google adds Cheminformatics Skills to Antigravity for chemistry workflow · 1 src
- AWS adds native on-behalf-of token exchange to AgentCore Gateway for ServiceNow · 1 src
Comments
via GitHub Discussions