AutomationBench joins Fireworks' Specialized Intelligence Index

The benchmark, built on patterns from over 2 billion monthly tasks across 3.7 million Zapier companies, tests whether an AI agent can complete end‑to‑end workflows in sales, marketing, operations, support, finance, and HR. It checks the actual outcome in each app rather than the agent’s answer, flagging failures that are hard to spot in live environments.

The leaderboard shows Gemini 4 Argon leading at 51.3 % of tasks, followed by Claude Sonnet 5.5 at 44.7 %. Even the best model fails about half the time, with false confidence being the most common failure mode. Cost varies widely: GPT‑6 Sol scores 80 % of GPT‑6 Astra for roughly a sixth of the cost, while GPT‑6 Luna and DeepSeek V4 Flash cost $0.04 per task but complete fewer than a quarter of full workflows.

For users, the benchmark suggests matching model choice to job complexity and verifying results, especially for customer‑facing or financial tasks.