Enterprise Agentic Retrieval in Amazon Bedrock with Observability

In a post published on the AWS blog by Luis Felipe Yepez Barrios, an enterprise agentic retrieval architecture is presented, built on Amazon Bedrock Managed Knowledge Bases and Amazon Bedrock AgentCore, alongside a seven-layer observability and evaluation setup deployed entirely through AWS CloudFormation.

According to the authors, teams integrating Retrieval Augmented Generation (RAG) mechanisms into foundation language models usually start with a single retrieval step against a single knowledge base. This approach works for simple questions, but when complex queries require cross-referencing information across multiple sources or deciding which source to consult, an agentic retrieval solution is required. In this model, an agent analyzes the question, routes it to the appropriate knowledge base, executes iterative retrieval queries, and returns a grounded answer accompanied by explicit citations. However, operating within an agentic loop where decisions and repeated retrievals take place makes it difficult to observe the steps taken by the agent and evaluate the quality of the final answer.

From Classic RAG to Enterprise Agentic Retrieval

In classic RAG, only a single retrieval step and a single generation step are performed. In contrast, enterprise agentic retrieval places a reasoning-capable agent in the loop. The agent decides whether and what to retrieve, can execute multiple retrievals to refine results, selects the relevant knowledge base using semantic routing, and only then drafts a grounded answer with citations. This capability is provided as a native feature in Amazon Bedrock Managed Knowledge Base through the AgenticRetrieveStream API.

The post notes that Amazon Bedrock now offers a Managed Knowledge Base (Type: MANAGED), where the Bedrock service fully manages document ingestion, storage, indexing, and retrieval, including embedding and reranking using service-managed models by default, without requiring the provisioning, scaling, or maintenance of a dedicated vector database.

Compared to a customer-managed (DIY) knowledge base, a Managed Knowledge Base supports agentic retrieval (AgenticRetrieveStream) and integration with the AgentCore Gateway, which are not supported in the customer-managed model. In addition, the Managed Knowledge Base includes an auto-scaling data store fully managed by Bedrock, built-in managed models for embedding and reranking (with the option to select other models available in Bedrock), and requires no vector database infrastructure management.

Solution Architecture and the Four CloudFormation Stacks

The solution is deployed using four chained AWS CloudFormation stacks:

1. Stack 01-knowledge-bases: Sets up an Amazon S3 bucket, two Managed Knowledge Bases (one for a financial corpus and one for a weather corpus), IAM settings, and data sources, as well as a custom resource to upload documents and trigger an initial sync.
2. Stack 02-agentic-gateway: Sets up the Amazon Bedrock AgentCore Gateway (based on AWS_IAM authentication and the Model Context Protocol - MCP) with a dedicated target for each knowledge base built on the native bedrock-knowledge-bases connector, allowing each knowledge base to expose an AgenticRetrieveStream tool without requiring an AWS Lambda function or an extra container.
3. Stack 03-agent-runtime: Provisions an Amazon ECR repository and an AWS CodeBuild project that builds an OpenTelemetry-instrumented Strands agent image, an Amazon Bedrock AgentCore runtime to host the agent, log and trace routing, and an online evaluation configuration.
4. Stack 04-dashboards: Creates two dashboards in Amazon CloudWatch.

Data flow occurs across two routing levels: the agent's reasoning model performs cross-knowledge-base routing, where each knowledge base has a dedicated retrieval tool and the agent selects the tool matching the question's topic. Next, the AgenticRetrieveStream API performs the within-knowledge-base work: it decomposes the question into sub-queries, retrieves information iteratively, and synthesizes a grounded, cited answer streaming back to the agent through the Gateway. The agent evaluates whether the returned information is sufficient, and if not, executes another retrieval iteration before composing the final answer for the user.

Seven Observability Layers and Two Dashboards

The runtime environment produces metrics and traces that feed into Amazon CloudWatch and AWS X-Ray, organized across seven observability layers:

• Layer 1 (L1 - KB-native metrics): Invocation, error, and throttle metrics per knowledge base, indicating service health and workload handling.
• Layer 2 (L2 - Ingestion): Ingestion job status and document processing results at the individual file level.
• Layer 3 (L3 - Agentic retrieval quality): Reference-free retrieval quality signals, including context utilization, grounded coverage, and duplicate rate. These metrics are defined without requiring chunk scores because the API returns a synthesized and cited answer.
• Layer 4 (L4 - Gateway / MCP metrics): Tool-call volume and latency at the Gateway.
• Layer 5 (L5 - OTEL span tree): A full trace tree of the agent's Reason-and-Act loop, displaying every step executed.
• Layer 6 (L6 - Token usage): Tracking of token consumption (gen_ai.usage) per session and model.
• Layer 7 (L7 - Evaluation scores): Evaluation scores covering Correctness, Faithfulness, Tool-Selection Accuracy, and Response Relevance.

Layers 1, 4, and 5 are emitted automatically, while Layers 3, 6, and 7 are published as custom metrics from the driver notebook. The system includes two dashboards: Dashboard A, which presents an end-to-end agentic observability view (for example, for N queries, approximately N agent invocations, 2N retrievals, 3N model calls, and 5N Gateway MCP operations are shown), and Dashboard B, which presents operational and spend-related data specific to each knowledge base (index size in bytes, retrieval volume, agentic calls, and token usage).

Quality Evaluation: On-Demand vs. Continuous

The solution provides two methods for evaluating answer quality:

1. On-Demand Evaluation: The driver notebook invokes AgentCore Evaluate using an LLM-as-judge across each session's traces for correctness, faithfulness, and tool-selection accuracy metrics, publishing the results to CloudWatch as Layer 7. This approach fits development and pre-release testing stages, where costs are incurred only for deliberate executions.
2. Continuous (Online) Evaluation: Stack 03 provisions an AWS::BedrockAgentCore::OnlineEvaluationConfig resource that samples live sessions and automatically computes scores within the CloudWatch console without running a notebook. In the post's example, a 100% sampling rate was set purely for demonstration purposes, but the authors emphasize that in production environments, the sampling percentage should be aligned with organizational budget and policies, as each sample invokes an LLM-as-judge and incurs additional costs.

Dataset, Prerequisites, and System Management

The dataset in the solution contains two separate corpora designed to demonstrate semantic routing: a synthetic financial corpus (Octank Financial 10-K report of approximately 198 KB) and a real weather corpus from a U.S. Congressional Research Service report on tornadoes (IF12695 of approximately 560 KB). During ingestion, Bedrock automatically performs parsing, chunking, embedding, and indexing without requiring manual parameter configuration.

Prerequisites include an AWS account with permissions for Bedrock, AgentCore runtime, AgentCore Gateway, IAM, CloudWatch, X-Ray, ECR, CodeBuild, S3, Lambda, and CloudFormation; access to the agent model (default us.anthropic.claude-haiku-4-5-20251001-v1:0); CloudWatch Transaction Search enabled; AWS CLI v2; and a Python 3.13 environment with boto3 version 1.43 or higher. Deployment is executed by running the deploy.sh script, and after use, all four stacks can be deleted in reverse dependency order using the cleanup.sh script.