Evaluating AI Agents: A production blueprint with Strands and AgentCore

<p><em>This post was co-written with Motorway and the AWS Prototyping and AI Customer Engineering (PACE) team.</em></p> <hr> <p>Motorway, a UK-based online car marketplace, runs a daily auction where up to 8,000 dealers bid on up to 2,500 vehicles. Motorway worked with AWS Prototyping and AI Customer Engineering (PACE) to build an AI-powered dealer stock search agent that transforms how dealers find vehicles, replacing hours of manual filtering with natural language queries.</p> <h3 id="the-challenge">The challenge</h3> <p>The agent gives a confident-sounding response, but how do you prove it works reliably with real money on the line?</p> <ul> <li><strong>Tool selection errors</strong> cause wrong search results, eroding dealer trust.</li> <li><strong>Semantic search misinterpretations</strong> return irrelevant results. A query like “Petrol, Hybrid and electric cars up to 5 years old” requires the agent to correctly parse multiple constraints.</li> <li><strong>Context drift</strong> in multi-turn conversations loses dealer refinements.</li> <li><strong>Non-deterministic outputs</strong> make single-trial testing unreliable.</li> </ul> <h3 id="the-solution">The solution</h3> <p>Together, Motorway and AWS built an end-to-end evaluation pipeline that reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection time from few hours to few minutes.</p> <p>The pipeline combines the Strands Agents SDK with Amazon Bedrock AgentCore, a fully managed service for deploying and operating AI agents at scale. In this post, you will learn how to build this pipeline for your own agents:</p> <ul> <li><strong>A two-phase evaluation strategy</strong> spanning build-time testing with <a href="https://github.com/strands-agents/evals" target="_blank" rel="noopener">strands-agents-evals</a> (the open source evaluation library for Strands Agents) and production monitoring with <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html" target="_blank" rel="noopener">Amazon Bedrock AgentCore Evaluations</a>.</li> <li><strong>A three-layer framework</strong> for assessing tool usage, reasoning, and output quality.</li> <li><strong>A five-stage deployment pipeline</strong> with quality gates that block releases when metrics fall below thresholds.</li> </ul> <p>A <a href="https://github.com/aws-samples/sample-evaluating-agents-on-aws-with-strands-and-agentcore" target="_blank" rel="noopener">companion repository</a> provides a deployable blueprint that you can adapt for your own agents. Although the blueprint uses AWS services, the core principles are essential and system-agnostic requirements for any production-ready AI agent. These principles include the three-layer evaluation framework and the use of the pass^k metric for consistency.</p> <h2 id="prerequisites">Prerequisites</h2> <p>You must have the following prerequisites to follow along with this post.</p> <ul> <li>An <a href="https://aws.amazon.com/free/" target="_blank" rel="noopener">AWS account</a> with permissions for <a href="https://aws.amazon.com/bedrock/" target="_blank" rel="noopener">Amazon Bedrock</a>, <a href="https://aws.amazon.com/lambda/" target="_blank" rel="noopener">AWS Lambda</a>, <a href="https://aws.amazon.com/s3/" target="_blank" rel="noopener">Amazon Simple Storage Service (Amazon S3)</a>, <a href="https://aws.amazon.com/dynamodb/" target="_blank" rel="noopener">Amazon DynamoDB</a>, <a href="https://aws.amazon.com/eventbridge/" target="_blank" rel="noopener">Amazon EventBridge</a>, <a href="https://aws.amazon.com/cloudwatch/" target="_blank" rel="noopener">Amazon CloudWatch</a>, and <a href="https://aws.amazon.com/sns/" target="_blank" rel="noopener">Amazon Simple Notification Service (Amazon SNS)</a>.</li> <li><a href="https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html" target="_blank" rel="noopener">AWS Cloud Development Kit (AWS CDK)</a> v2 installed and bootstrapped.</li> <li>