How Do You Evaluate an AI Agent? Tasks, Trajectories, Reliability, and Cost

The guide proposes four separate evaluation layers: outcome quality (did the agent achieve the goal?), trajectory quality (did it take a sound path?), safety and policy compliance (did it stay within authority?), and operational efficiency (what was the cost in time, tokens, and retries?). Teams should use multiple types of graders—deterministic, model-based, and human—to avoid bias. Reliability demands repeated trials, with metrics like Pass@k (probability of success within k attempts) or Pass^k (consistency across all attempts). The guide also warns against common mistakes, such as ignoring trajectories, relying on single runs, or overlooking data leakage. Agents’ performance depends on their environment, so versioning the entire stack (model, tools, data) is critical.