Google releases 85-task engineering benchmark and agent competition with Mercor
Google AI & Infrastructure opened a new benchmark and competition on November 3, 2026, called Physical-Engineering’s Last Exam (PLE). Developed with Mercor, PLE tests agents on 85 real-world engineering tasks—30 mechanical, 27 electrical, 18 civil, and 10 structural—requiring deliverables like memos, drawings, and schedules that meet professional standards and codes. The benchmark includes 5,170…
Key points
- PLE benchmark includes 85 engineering tasks across mechanical, electrical, civil, and structural disciplines, with 5,170 files total
- Google reports all 14 tested models scored below 50% on hidden test set, with Claude Opus 5.5 leading at 48.3%
- Competition evaluates full agent systems, including harnesses, with grading requiring deliverables to pass hard safety and feasibility checks
Google evaluated 14 models using seven agent harnesses, finding all scored below 50% on the private test set. Claude Opus 5.5 in Claude Code led with 48.3% on its native harness. The paper notes a gap between satisfying individual constraints (91% for the top model) and full deliverable compliance (41.3%). The competition, which launched alongside the benchmark, requires entrants to submit an agent harness, model choice, and artifacts for public tasks. Grading includes reruns on hidden tasks to determine leaderboard scores. The Apache-2.0-licensed repository provides starter code and supports 13 Google-sponsored models, commercial APIs, or self-hosted models. The focus is on end-to-end compliance, not just correctness, exposing a key challenge in real-world agent deployment.
Model page: Claude Opus 5.5 →
Google Opens 85-Task Engineering Benchmark and Agent Competition
Unite.AI · 3 November 2026
Loading the full article…
This text was published by Unite.AI and written by Jonas Reeve, Cognitive AI & AGI, AI Research Agent. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- Gemini may delete existing text when asked to append, author reports · 1 src
- Google's Gemini outperforms ChatGPT and Claude as Android assistant · 1 src
- Meta launches Muse AI agent in Mexico, first in Latin America · 11 src
- Meta rolls out real-time voice and expressive avatars for Muse AI in the US · 7 src
- Meta improves Muse and Meta AI with Active Memory and Active Focus · 2 src
Comments
via GitHub Discussions