# Amazon Bedrock adds decision models for agent evaluation

Digest AI · Agents & Tools · published 2026-11-04T18:46:36Z

Canonical: https://digestai.news/story/amazon-bedrock-adds-decision-models-for-agent-evaluation

## Summary

Amazon Bedrock AgentCore Evaluations now supports decision models as custom evaluators for agent performance checks. These models—like **Jev** (hosted by TypeSafe) and **Strands Decider 2B** (open-source, 2B parameters)—score agent outputs with calibrated probabilities over predefined options (yes/no, choice, or score) instead of free-form text. They’re designed for narrow, repeatable checks (e.g., policy compliance, fact grounding) where a binary label and confidence metric are needed, not deep explanations. The blog demonstrates integrating these models via AWS Lambda functions into AgentCore’s evaluation pipeline, alongside built-in LLM-as-a-judge evaluators for broader assessments.

The approach treats decision models as a screening layer: they flag potential failures cheaply, while built-in evaluators handle deeper analysis. AWS provides a **Python-based HR Assistant agent example** with mock HR data (PTO balances, policies) to test both decision-model and LLM evaluators. Key steps include validating the models against labeled human reference labels, comparing agreement rates, and calibrating confidence thresholds. The blog warns that reframing rubrics as decision-model questions may shift strictness and advises treating validation as an ongoing process. Prerequisites include an AWS account, Python 3.12+, and access to TypeSafe’s API (for **Jev**) or a self-hosted Strands Decider server.

## Key points

- Amazon Bedrock AgentCore Evaluations now supports **Jev** (TypeSafe) and **Strands Decider 2B** (open-source, 2B params) as custom evaluators for agent scoring
- Decision models return calibrated probabilities over fixed options (yes/no, choice, or score) for narrow checks like policy compliance or fact validation
- AWS provides a **Python HR Assistant demo** with mock data to compare decision-model and LLM evaluators, emphasizing validation against human-labeled traffic

## Why it matters

Decision models add deterministic, thresholdable checks to agent evaluation pipelines, complementing LLM-based evaluators for tasks needing precise scoring. This reduces reliance on ambiguous text-based judgments and enables automated gating of agent outputs, critical for production systems where explainability and consistency matter.

## Sources

1. [Use decision models with Amazon Bedrock AgentCore Evaluations](https://aws.amazon.com/blogs/machine-learning/use-decision-models-with-amazon-bedrock-agentcore-evaluations) (AWS Machine Learning Blog, 2026-11-04, primary source)

Part of the developing story: [AWS Builds Foundation for Production AI Agents](https://digestai.news/thread/aws-blog-shows-how-to-build-agentic-ai-semantic-layer-with-edb-postgres-ai) (4 stories)

## Cite

Digest AI, "Amazon Bedrock adds decision models for agent evaluation", 4 November 2026, https://digestai.news/story/amazon-bedrock-adds-decision-models-for-agent-evaluation

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/amazon-bedrock-adds-decision-models-for-agent-evaluation.json