Amazon adds skill selection and instruction-following evaluators to Bedrock AgentCore
Amazon Web Services announced that the Strands Evals SDK and Amazon Bedrock AgentCore now include skill‑focused evaluators. The new metrics—Skill Selection Accuracy and Skill Instruction Following—measure whether an agent picks the right reusable skill and how completely it follows the skill’s prescribed steps. A deterministic check called SkillInvoked confirms that a named skill was loaded…
Key points
- Strands Evals SDK and Bedrock AgentCore now offer Skill Selection Accuracy and Skill Instruction Following evaluators.
- Skill Instruction Following rates execution on a five‑level scale from Fully Followed (1.0) to Not Followed (0.0).
- Evaluations can run on recorded trajectories, OpenTelemetry traces, or continuously on live traffic via the AgentCore CLI.
The evaluators work on recorded agent trajectories in Strands Evals or on OpenTelemetry traces captured by AgentCore. Skill Instruction Following rates execution on a five‑level scale (Fully Followed 1.0, Mostly Followed 0.75, Partially Followed 0.5, Minimally Followed 0.25, Not Followed 0.0). Developers can run these checks in unit‑test pipelines, batch over stored sessions, or continuously on live traffic using the AgentCore CLI. Prerequisites include Python 3.10+, an AWS account with Bedrock access, and enabled CloudWatch observability.
The blog post provides step‑by‑step examples for an HR‑assistant agent, guidance on interpreting per‑skill results, and best practices for combining deterministic and judge‑based metrics. It emphasizes keeping routing and execution failures separate to diagnose issues quickly and to block releases when skill regressions are detected.
Evaluate skill-equipped agents with Strands Evals and Amazon Bedrock AgentCore
AWS Machine Learning Blog · 22 September 2026
Loading the full article…
This text was published by AWS Machine Learning Blog and written by Sangmin Woo. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- OpenAI launches Dots agent for ChatGPT Pro and Business Premium · 4 src
- Claude Code builds a file-organizing AI agent in 30 minutes · 1 src
- Anthropic lets users mod Claude Code like video game plugins · 2 src
- Cresta builds Conductor agent builder on Claude Agent SDK · 1 src
- GitHub launches ReviewBench to test AI code review agents · 3 src
Comments
via GitHub Discussions