Amazon Bedrock adds decision models for agent evaluation
Amazon Bedrock AgentCore Evaluations now supports decision models as custom evaluators for agent performance checks. These models—like Jev (hosted by TypeSafe) and Strands Decider 2B (open-source, 2B parameters)—score agent outputs with calibrated probabilities over predefined options (yes/no, choice, or score) instead of free-form text. They’re designed for narrow, repeatable checks (e.g.,…
Key points
- Amazon Bedrock AgentCore Evaluations now supports **Jev** (TypeSafe) and **Strands Decider 2B** (open-source, 2B params) as custom evaluators for agent scoring
- Decision models return calibrated probabilities over fixed options (yes/no, choice, or score) for narrow checks like policy compliance or fact validation
- AWS provides a **Python HR Assistant demo** with mock data to compare decision-model and LLM evaluators, emphasizing validation against human-labeled traffic
The approach treats decision models as a screening layer: they flag potential failures cheaply, while built-in evaluators handle deeper analysis. AWS provides a Python-based HR Assistant agent example with mock HR data (PTO balances, policies) to test both decision-model and LLM evaluators. Key steps include validating the models against labeled human reference labels, comparing agreement rates, and calibrating confidence thresholds. The blog warns that reframing rubrics as decision-model questions may shift strictness and advises treating validation as an ongoing process. Prerequisites include an AWS account, Python 3.12+, and access to TypeSafe’s API (for Jev) or a self-hosted Strands Decider server.
Model pages: Jev → · Strands Decider 2B →
The story so far
4 episodes →- Amazon Bedrock adds decision models for agent evaluationthis story
Use decision models with Amazon Bedrock AgentCore Evaluations
AWS Machine Learning Blog · 4 November 2026
Loading the full article…
This text was published by AWS Machine Learning Blog and written by Sungyeon Kim. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- Datasette Agent adds panel feature for database-backed app editing · 1 src
- Anthropic’s Claude Duet adds always-on agent to Pro, Max, Ultra plans · 1 src
- AWS details four context engineering layers for production AI agents · 1 src
- How Do You Evaluate an AI Agent? Tasks, Trajectories, Reliability, and Cost · 1 src
- Apple opens Siri AI to developers via iOS 27.3 betas · 3 src
Comments
via GitHub Discussions