In a post published on the AWS blog, a cloud-based solution featuring a smart caching mechanism was presented, designed to capture, maintain, and deliver accumulated institutional knowledge through an intelligent avatar system running on AWS services. According to the post's authors, organizations across diverse industries struggle with managing institutional knowledge and experience accumulated over years of operations, which is often lost when key employees leave. Traditional documentation methods frequently prove inadequate, leaving outdated or inaccessible information.
Target Audiences and System Use Cases
According to the post, organizations across diverse industries can use the system to preserve critical knowledge. In manufacturing organizations, for example, production procedures and maintenance protocols can be captured before experienced technicians retire. Additionally, healthcare facilities, financial services firms, energy companies, and government agencies can tailor the solution to their needs. Knowledge workers can access procedures and policies through natural language queries instead of searching multiple repositories, while subject matter experts or retiring employees can upload documentation to preserve their expertise for future generations.
The system can be deployed with desktop browser access for detailed research, voice interaction for hands-free operation, and text-based queries for quick reference depending on the operational context of each industry.
Solution Architecture and Cloud Components
At the foundation of the solution is a browser-based interface supporting both voice and text interactions, connecting to a configurable avatar system that works with third-party avatar solutions (implemented here based on DeepBrain AI).
Behind the scenes, the following AWS services operate:
- Amazon Cognito secures access management and user controls.
- Amazon API Gateway provides managed, monitored endpoints for communication between system components.
- Amazon Bedrock Knowledge Bases serves as the knowledge-processing core for managed Retrieval Augmented Generation (RAG): data stored in Amazon S3 serves as the data source, and Bedrock performs chunking, vector generation (embedding) using the Amazon Titan Text Embeddings model, and document-grounded retrieval.
- Amazon OpenSearch Serverless serves as a vector store for the Bedrock knowledge base.
- Amazon DynamoDB provides a response caching layer.
- AWS Lambda functions orchestrate request and response workflows.
- Amazon Transcribe converts voice input to text.
- Amazon Polly converts text responses to natural speech.
- Avatar video streaming operates over WebRTC alongside WebSocket connections for response control.
Regarding costs, the post noted that the OpenSearch Serverless vector store is billed per OpenSearch Compute Unit (OCU) with an always-on minimum threshold independent of query volume, representing a fixed baseline cost of a few hundred dollars per month in the default configuration, on top of which are variable language model costs that the caching mechanism is designed to reduce.
Implementation Phases of the Solution
The setup process is described in three main phases:
- Knowledge Foundation Setup: Documents are uploaded to a repository in Amazon S3 in supported formats such as Word, PDF, plain text, Markdown, or JSON (with Markdown and JSON recommended for optimal retrieval performance). An AWS Glue ETL job can optionally be used to convert documents from other formats before uploading.
- Infrastructure Deployment: Infrastructure deployment using AWS CloudFormation templates, which includes setting up Amazon Cognito for user authentication and access controls, configuring API Gateway endpoints, establishing S3 repositories for knowledge storage, and implementing DynamoDB for response caching.
- AI Integration: Configuring Amazon Bedrock, connecting Lambda functions for orchestration, setting up the audio processing pipeline with Transcribe and Polly, and integrating the avatar system.
Initial deployment requires appropriate AWS IAM permissions for all involved services. The foundation models used in the implementation are Amazon Titan Text Embeddings for document encoding, as well as Amazon Nova Pro and Anthropic Claude 3 Sonnet as selectable models for generating responses. According to the post, Amazon Bedrock provides automatic access to Amazon-owned serverless models, so Amazon Nova Pro and Amazon Titan Text Embeddings are available by default with no manual enablement; conversely, Anthropic Claude models are not covered by automatic access in all Regions, and before the first Claude model request, a one-time model-access request must be completed in the Bedrock console and appropriate AWS Marketplace subscription permissions ensured.
Performance Optimization and the Caching Mechanism
The system implements a dual-layer caching strategy:
- A browser-side cache using an LRU algorithm in browser memory for instant responses to repeated questions.
- A server-side cache in DynamoDB with intelligent Time-To-Live (TTL) settings tailored to content type: longer retention for fundamental knowledge, medium-term for operational procedures, and short-term for time-sensitive information.
In the current implementation, cache matching is performed based on exact query text matching. In AWS testing, for workloads with 50 to 70 percent repeated questions, the cache hit rate resulted in a comparable reduction in variable language model inference costs.
Knowledge Updates, Accuracy Considerations, and Limitations
To keep the system current, Amazon S3 event notifications are configured: adding or removing a document triggers a Lambda function that initiates synchronization and re-indexing in Bedrock Knowledge Bases without manual intervention.
In terms of accuracy, Bedrock grounds answers in passages retrieved from the organization's verified documents and can return source citations for verification. However, the post emphasizes that this grounding reduces but does not completely eliminate the risk of incorrect answers, so for high-consequence or safety-related decisions, a human should be kept in the loop and the system should be treated as decision support rather than an exclusive authoritative source.
Additionally, this architecture depends on continuous cloud connectivity and includes no offline capability in this prototype, making it suitable for connected environments (such as training rooms, quality labs, and control rooms) rather than network-disconnected plant floors. In testing, the default configuration was estimated to support roughly 50 to 100 concurrent users, with the figures representing indicative estimates only and not guarantees, depending on Region, model choice, document size, cache hit rate, and service quotas; enterprise-scale expansion to over 5,000 concurrent users is achievable with the same architecture but requires proactive service-quota increases.