Build custom code-based evaluators in Amazon Bedrock AgentCore

🔑 Enhanced Key Takeaways

•Custom code evaluators in Amazon Bedrock AgentCore allow for deterministic checks, external API calls, regex matching, and custom metric computation, offering full control over evaluation logic beyond traditional LLM-as-a-judge methods.
•Beyond custom evaluators, AgentCore Evaluations provides 13 pre-built evaluators covering essential quality dimensions such as correctness, helpfulness, tool selection accuracy, safety, and goal success rate.
•Evaluations can be configured to operate at granular levels, including session-level (entire conversations), trace-level (individual responses), or tool-call level (specific tool usage), enabling precise diagnosis of agent behavior.
•Amazon Bedrock AgentCore is designed to be framework-agnostic and model-agnostic, supporting integration with popular open-source frameworks like CrewAI, LangGraph, and LlamaIndex, and various foundation models from different providers, including OpenAI, Google's Gemini, and Anthropic's Claude.
•Pricing for custom code evaluators is consumption-based at $1.50 per 1,000 evaluations, which is distinct from the token-based pricing for built-in LLM-as-a-judge evaluators.

📊 Competitor Analysis▸ Show

While Amazon Bedrock AgentCore provides a comprehensive platform for building and evaluating AI agents, several specialized platforms offer competitive features in the evaluation space:

Feature / Platform	Amazon Bedrock AgentCore Evaluations	Maxim AI	LangSmith	Langfuse
Evaluation Types	Built-in (LLM-as-a-Judge), Custom Code (Lambda), Ground Truth	Pre-built, Custom (deterministic, statistical, LLM-based), Conversation-level	Scenario-based, Batch scoring, RAG support	Rule-based, LLM judge custom metrics, Human labeling
Observability	Integrated with CloudWatch, detailed visualizations of agent workflow, OpenTelemetry/OpenInference instrumentation	End-to-end lifecycle coverage, connects pre-production experimentation with post-deployment monitoring	Tracing, monitoring, prompt comparisons	Real-time traces, prompt comparisons, integrations
Integration/Flexibility	Framework-agnostic (CrewAI, LangGraph, LlamaIndex), Model-agnostic (OpenAI, Gemini, Claude, etc.), Lambda for custom logic	Full-stack approach for multi-agent systems, Playground++ for prompt iteration	Deep LangChain integration	Open-source, MIT-licensed, self-hosting supported, integrates with LangChain
Deployment Model	Fully managed service	Managed service, with open-source gateway	Managed service	Open-source, self-hosting available
Primary Focus	Securely deploy, operate, and continuously evaluate AI agents at scale with custom and built-in tools	End-to-end AI quality platform, simulation, evaluation, observability	Workflow testing, debugging, and monitoring for LangChain applications	Tracing, evaluation, and prompt management for LLM applications

🛠️ Technical Deep Dive

Custom code evaluators are implemented as AWS Lambda functions.
These Lambda functions receive OpenTelemetry (OTel) spans as structured events, which contain detailed information about the agent's reasoning, tool calls, and responses.
The Lambda function is expected to return a JSON response that includes a score, a label, and an explanation for the evaluation.
AgentCore Runtime, which hosts the AI agents and tools, operates in a secure, serverless environment, similar to AWS Lambda, and packages agent code into Docker images for deployment to Amazon ECR.
Each user session within AgentCore Runtime runs in its own dedicated microVM, providing complete session isolation for enhanced security and data protection.
AgentCore supports OpenTelemetry and OpenInference instrumentation libraries for collecting detailed traces of agent interactions.
The evaluation process can be applied at three levels: session (entire conversation), trace (individual turns/responses), and tool-call (specific tool invocations), with varying contextual information available at each level for the evaluators.
AgentCore Gateway facilitates secure access to tools by transforming existing APIs and Lambda functions into agent-compatible tools using the Model Context Protocol (MCP).

🔮 Future ImplicationsAI analysis grounded in cited sources

Enterprises will increasingly adopt complex, multi-step AI agents for critical business operations.

The ability to implement custom, deterministic evaluators, combined with comprehensive monitoring and security features, will build greater trust and confidence in deploying sophisticated AI agents in production environments.

The demand for specialized AI agent evaluation and observability platforms will continue to grow.

As AI agents become more prevalent and complex, the need for robust tools to ensure their quality, safety, and performance, including both built-in and highly customizable evaluation methods, will become paramount.

AI agent development will shift towards more auditable and transparent workflows.

The granular evaluation capabilities at session, trace, and tool-call levels, coupled with custom code evaluators for deterministic checks, will enable developers to create agents with clearer accountability and easier debugging.

⏳ Timeline

2023-04

Amazon Bedrock announced in preview.

2023-09

Amazon Bedrock General Availability.

2025-07

Amazon Bedrock AgentCore announced in preview.

2025-10

Amazon Bedrock AgentCore General Availability.

2025-12

AgentCore Evaluations (including custom evaluators) launched in preview.

2026-04

Code-based evaluators using AWS Lambda became available in AgentCore Evaluations.

Build custom code-based evaluators in Amazon Bedrock AgentCore

⚡ 30-Second TL;DR

Key Points

🧠 Deep Insight

🔑 Enhanced Key Takeaways

🛠️ Technical Deep Dive

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline

📎 Sources (22)

👉Related Updates

OpenAI agent escaped due to preventable human errors

Anthropic and OpenAI disclose AI systems breaching external networks

GM triples pull requests by redesigning workflows around AI agents

First Agentic Diffusion Model with 128K Context