Amazon adds skill selection and instruction-following evaluators to Bedrock AgentCore
Amazon Web Services announced that the Strands Evals SDK and Amazon Bedrock AgentCore now include skill‑focused evaluators. The new metrics—Skill Selection Accuracy and Skill Instruction Following—measure whether an agent picks the right reusable skill and how completely it follows the skill’s prescribed steps. A deterministic check called SkillInvoked confirms that a named skill was loaded…
Key points
- Strands Evals SDK and Bedrock AgentCore now offer Skill Selection Accuracy and Skill Instruction Following evaluators.
- Skill Instruction Following rates execution on a five‑level scale from Fully Followed (1.0) to Not Followed (0.0).
- Evaluations can run on recorded trajectories, OpenTelemetry traces, or continuously on live traffic via the AgentCore CLI.
The evaluators work on recorded agent trajectories in Strands Evals or on OpenTelemetry traces captured by AgentCore. Skill Instruction Following rates execution on a five‑level scale (Fully Followed 1.0, Mostly Followed 0.75, Partially Followed 0.5, Minimally Followed 0.25, Not Followed 0.0). Developers can run these checks in unit‑test pipelines, batch over stored sessions, or continuously on live traffic using the AgentCore CLI. Prerequisites include Python 3.10+, an AWS account with Bedrock access, and enabled CloudWatch observability.
The blog post provides step‑by‑step examples for an HR‑assistant agent, guidance on interpreting per‑skill results, and best practices for combining deterministic and judge‑based metrics. It emphasizes keeping routing and execution failures separate to diagnose issues quickly and to block releases when skill regressions are detected.
Evaluate skill-equipped agents with Strands Evals and Amazon Bedrock AgentCore
AWS Machine Learning Blog · 22 September 2026
Loading the full article…
This text was published by AWS Machine Learning Blog and written by Sangmin Woo. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- Nvidia releases Isaac ROS 5.0 with agentic workflows and ROS Lyrical support · 2 src
- Claude Code adds support for OpenAI Agents.md format · 4 src
- Meta tests Muse calls that are actually made by humans in a call center · 1 src
- Claude Code stops working, user fixes it with Google's Antigravity (Gemini) · 1 src
- Simon Willison releases llm-typesafe plugin for TypeSafe AI's Jev model · 1 src
Comments
via GitHub Discussions