{"version":1,"type":"story","url":"https://digestai.news/story/amazon-bedrock-adds-decision-models-for-agent-evaluation","json":"https://digestai.news/story/amazon-bedrock-adds-decision-models-for-agent-evaluation.json","markdown":"https://digestai.news/story/amazon-bedrock-adds-decision-models-for-agent-evaluation.md","slug":"amazon-bedrock-adds-decision-models-for-agent-evaluation","headline":"Amazon Bedrock adds decision models for agent evaluation","summary":"Amazon Bedrock AgentCore Evaluations now supports decision models as custom evaluators for agent performance checks. These models—like **Jev** (hosted by TypeSafe) and **Strands Decider 2B** (open-source, 2B parameters)—score agent outputs with calibrated probabilities over predefined options (yes/no, choice, or score) instead of free-form text. They’re designed for narrow, repeatable checks (e.g., policy compliance, fact grounding) where a binary label and confidence metric are needed, not deep explanations. The blog demonstrates integrating these models via AWS Lambda functions into AgentCore’s evaluation pipeline, alongside built-in LLM-as-a-judge evaluators for broader assessments.\n\nThe approach treats decision models as a screening layer: they flag potential failures cheaply, while built-in evaluators handle deeper analysis. AWS provides a **Python-based HR Assistant agent example** with mock HR data (PTO balances, policies) to test both decision-model and LLM evaluators. Key steps include validating the models against labeled human reference labels, comparing agreement rates, and calibrating confidence thresholds. The blog warns that reframing rubrics as decision-model questions may shift strictness and advises treating validation as an ongoing process. Prerequisites include an AWS account, Python 3.12+, and access to TypeSafe’s API (for **Jev**) or a self-hosted Strands Decider server.","keyPoints":["Amazon Bedrock AgentCore Evaluations now supports **Jev** (TypeSafe) and **Strands Decider 2B** (open-source, 2B params) as custom evaluators for agent scoring","Decision models return calibrated probabilities over fixed options (yes/no, choice, or score) for narrow checks like policy compliance or fact validation","AWS provides a **Python HR Assistant demo** with mock data to compare decision-model and LLM evaluators, emphasizing validation against human-labeled traffic"],"whyItMatters":"Decision models add deterministic, thresholdable checks to agent evaluation pipelines, complementing LLM-based evaluators for tasks needing precise scoring. This reduces reliance on ambiguous text-based judgments and enables automated gating of agent outputs, critical for production systems where explainability and consistency matter.","category":{"slug":"agents","name":"Agents & Tools","url":"https://digestai.news/category/agents"},"entities":{"companies":["Amazon","TypeSafe","AWS"],"models":["Jev","Strands Decider 2B"],"people":[]},"firstPublishedAt":"2026-11-04T18:46:36Z","updatedAt":"2026-11-04T18:46:36Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"AWS Machine Learning Blog","title":"Use decision models with Amazon Bedrock AgentCore Evaluations","url":"https://aws.amazon.com/blogs/machine-learning/use-decision-models-with-amazon-bedrock-agentcore-evaluations","publishedAt":"2026-11-04T18:46:36Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":{"title":"AWS Builds Foundation for Production AI Agents","url":"https://digestai.news/thread/aws-blog-shows-how-to-build-agentic-ai-semantic-layer-with-edb-postgres-ai","storyCount":4},"cite":{"text":"Digest AI, \"Amazon Bedrock adds decision models for agent evaluation\", 4 November 2026, https://digestai.news/story/amazon-bedrock-adds-decision-models-for-agent-evaluation","publisher":"Digest AI","title":"Amazon Bedrock adds decision models for agent evaluation","datePublished":"2026-11-04T18:46:36Z","url":"https://digestai.news/story/amazon-bedrock-adds-decision-models-for-agent-evaluation"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}