# Rag App On Aws > Build and deploy a full-stack RAG app on AWS with Terraform, using free tier Gemini Pro, real-time web search using Remote MCP server and Streamlit UI with token based authentication. - **Type:** MCP server - **Install:** `agentstack add mcp-genieincodebottle-rag-app-on-aws` - **Verified:** Yes β€” security-reviewed for prompt injection and unsafe behavior - **Seller:** [genieincodebottle](https://agentstack.voostack.com/s/genieincodebottle) - **Installs:** 0 - **Category:** [Databases](https://agentstack.voostack.com/c/databases) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [genieincodebottle](https://github.com/genieincodebottle) - **Source:** https://github.com/genieincodebottle/rag-app-on-aws - **Website:** https://aimlcompanion.ai/ ## Install ```sh agentstack add mcp-genieincodebottle-rag-app-on-aws ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About     πŸ‘‰ GenAI Roadmap - 2025 ### End-to-End RAG App with Evaluation on AWS, Integrating Web Search via Remote MCP Server Terraform-based **Infrastructure as Code (IaC)** to deploy a complete AWS backend for a Retrieval-Augmented Generation (RAG) application. It integrates with Google’s free-tier Gemini Pro and Embedding models for AI powered document querying and includes a Streamlit UI with token-based authentication for interacting with the app. πŸ‘‰ Related Remote MCP Server: [Web Search using SerpAPI](https://github.com/genieincodebottle/rag-app-on-aws/tree/main/mcp_servers) Remote MCP Server based on Streaming Http Transport protocol for Real Time Web Search. It's located within the `mcp_servers/` directory of this repository. πŸ‘‰ Related UI: [RAG UI (Streamlit Frontend)](https://github.com/genieincodebottle/rag-app-on-aws/tree/main/rag_ui) A Streamlit-based frontend application designed to interact with the backend infrastructure deployed by this project. It's located within the `rag_ui/` directory of this repository. πŸ’° Estimated cost: ~$3 (~β‚Ή250) to experiment without the AWS Free Tier, primarily for RDS and NAT Gateway if active. πŸŽ₯ YouTube Video: Walkthrough on setting up the application, building, deploying, and running it end-to-end πŸ‘‡ [](https://www.youtube.com/watch?v=x2P4Ee6PYNg) --- ### πŸ” Overview This repository contains the complete Terraform codebase for provisioning and managing the AWS infrastructure that powers a RAG application. It allows users to upload documents, which are then processed, embedded, and stored for efficient semantic search and AI-driven querying. πŸ“Œ Key features include: - **IaC with Terraform**: For consistent and repeatable deployments across environments. - **Serverless Compute**: AWS Lambda for backend logic (document processing, querying, uploads, authentication, DB initialization). - **Vector Storage**: PostgreSQL RDS with the `pgvector` extension for storing and searching text embeddings. - **AI Integration**: Leverages Google's Gemini Pro (for generation) and Gemini Embedding models (for text embeddings). - **Authentication**: Secure user management with AWS Cognito. - **CI/CD Workflows**: GitHub Actions for automated deployment, testing, and cleanup. - **Multi-Environment Support**: Designed for `dev`, `staging`, and `production` environments. - **Comprehensive Testing**: Includes unit and integration tests for backend Lambda functions. - **Streamlit UI**: Includes a login page, document upload, query interface, and RAG evaluation dashboard. --- ### πŸ—οΈ High Level Architecture --- ### 🌐 Network Flow Walkthrough (Referencing the Architecture) #### πŸ—‚οΈ Document Processing Flow with Network Components: 1. User uploads document β†’ API Gateway β†’ `upload_handler` Lambda 2. `upload_handler` Lambda β†’ S3 Gateway Endpoint β†’ S3 Bucket 3. S3 Event β†’ `document_processor` Lambda (in private subnet) 4. `document_processor` Lambda β†’ NAT Gateway β†’ Internet Gateway β†’ Gemini API (for embeddings) 5. `document_processor` Lambda β†’ RDS Security Group β†’ PostgreSQL Database (stores chunks/vectors) #### πŸ’¬ Query Processing Flow with Network Components: 1. User submits query β†’ API Gateway β†’ `query_processor` Lambda (in private subnet) 2. `query_processor` Lambda β†’ RDS Security Group β†’ PostgreSQL Database (vector search) 3. `query_processor` Lambda β†’ NAT Gateway β†’ Internet Gateway β†’ Gemini API (for answer generation) 4. `query_processor` Lambda β†’ API Gateway β†’ User (returns answer) This network architecture ensures that sensitive operations and data are processed in a secure environment, while still allowing the necessary external communications through controlled channels. --- ### πŸ” GitHub Action Pipeline --- ### πŸ” AWS Infra Provisioning Flow Diagram πŸ—ΊοΈ [Infra Provisioning Lifecycle Flow](https://github.com/genieincodebottle/rag-app-on-aws/blob/main/images/infra_provisioning_sequence.png) (Illustrates the Terraform provisioning sequence) --- ### πŸ—‚οΈ Repository Structure ``` . β”œβ”€β”€ .github/workflows/ # CI/CD via GitHub Actions β”‚ β”œβ”€β”€ deploy.yml # Infrastructure deployment workflow β”‚ └── manual_cleanup.yml # Resource cleanup workflow β”œβ”€β”€ environments/ # Environment-specific configs (dev, staging, prod) β”‚ └── dev/ # Example 'dev' environment β”‚ β”œβ”€β”€ main.tf # Root Terraform file for the environment β”‚ β”œβ”€β”€ providers.tf # Terraform provider configurations β”‚ └── variables.tf # Environment-specific variable definitions β”œβ”€β”€ modules/ # Reusable Terraform modules β”‚ β”œβ”€β”€ api/ # API Gateway configuration β”‚ β”œβ”€β”€ auth/ # Cognito authentication β”‚ β”œβ”€β”€ compute/ # Lambda functions & IAM roles β”‚ β”œβ”€β”€ database/ # PostgreSQL RDS with pgvector & Secrets Manager β”‚ β”œβ”€β”€ monitoring/ # CloudWatch Logs, Alarms & SNS Topic β”‚ β”œβ”€β”€ storage/ # S3 Buckets & DynamoDB Table β”‚ └── vpc/ # VPC, Subnets, NAT, Security Groups, Endpoints β”œβ”€β”€ rag_ui/ # Streamlit UI application β”‚ β”œβ”€β”€ app.py # Main Streamlit application code β”‚ └── README.md # README specific to the UI β”œβ”€β”€ scripts/ # Utility shell scripts β”‚ β”œβ”€β”€ cleanup.sh # Comprehensive resource cleanup script β”‚ β”œβ”€β”€ import_resources.sh # Script to import existing AWS resources into Terraform state β”‚ └── network-diagnostics.sh # Script for troubleshooting network connectivity (e.g., Lambda to RDS) β”œβ”€β”€ src/ # Lambda backend source code (Python) β”‚ β”œβ”€β”€ auth_handler/ # Lambda for Cognito authentication operations β”‚ β”œβ”€β”€ db_init/ # Lambda for database schema and pgvector initialization β”‚ β”œβ”€β”€ document_processor/ # Lambda for processing uploaded documents β”‚ β”œβ”€β”€ query_processor/ # Lambda for handling user queries and RAG β”‚ β”œβ”€β”€ tests/ # Unit and integration tests β”‚ β”‚ β”œβ”€β”€ integration/ # Integration tests for deployed services β”‚ β”‚ β”‚ └── run_integration_tests.py β”‚ β”‚ β”œβ”€β”€ unit/ # Unit tests for Lambda functions β”‚ β”‚ β”‚ β”œβ”€β”€ conftest.py # Pytest common fixtures and mocks β”‚ β”‚ β”‚ β”œβ”€β”€ test_*.py # Individual unit test files β”‚ β”‚ └── __init__.py β”‚ β”œβ”€β”€ upload_handler/ # Lambda for handling file uploads via API β”‚ └── utils/ # Shared utility code (e.g., db_connectivity_test.py) β”œβ”€β”€ sonar-project.properties # SonarQube configuration file └── tox.ini # tox configuration for running tests and linters ``` --- ### 🧱 Infrastructure Components The infrastructure is modularized using Terraform modules: #### 1. **Networking (VPC - `modules/vpc`)** - Custom VPC with public, private, and database subnets across multiple Availability Zones. - Internet Gateway for public subnet access. - NAT Gateways (configurable for single or multiple AZs) for private subnet outbound access. - Route Tables for managing traffic flow. - Security Groups to control access to Lambdas, RDS, and Bastion hosts. - VPC Endpoints for S3 and DynamoDB, allowing private access from within the VPC. - Optional VPC Flow Logs for network traffic monitoring (enabled for `prod`). #### 2. **Compute (Lambda Functions - `modules/compute`, `src/`)** - All functions are Python 3.11 based. - **Authentication Handler (`auth_handler`)**: Manages user authentication lifecycle with Cognito (registration, login, email verification, password reset, token refresh). - **Document Processor (`document_processor`)**: - Triggered by S3 uploads to the `uploads/` prefix in the documents bucket. - Downloads the uploaded file (PDF, TXT, CSV, etc.). - Loads and chunks the document content. - Generates text embeddings for chunks using the Gemini Embedding model. - Stores document metadata and text chunks (with embeddings) in the PostgreSQL RDS database. - **Query Processor (`query_processor`)**: - Handles user queries from the API. - Generates an embedding for the user's query using the Gemini Embedding model. - Performs a vector similarity search in PostgreSQL (using `pgvector`) against stored document chunks. - Retrieves relevant chunks and prepares a context. - Generates a final answer using the Gemini Pro model with the retrieved context. - Optionally performs RAG evaluation (faithfulness, relevancy, context precision). - **Upload Handler (`upload_handler`)**: - API endpoint for initiating file uploads. - Receives file content (base64 encoded), name, and user ID. - Uploads the raw file to a specific S3 path (`uploads/{user_id}/{document_id}/{file_name}`). - Stores initial document metadata in PostgreSQL and DynamoDB. - **DB Initialization (`db_init`)**: - A Lambda function invoked during CI/CD deployment. - Connects to the PostgreSQL RDS instance. - Creates necessary database tables (`documents`, `chunks`) if they don't exist. - Enables the `pgvector` extension required for vector operations. - **IAM Roles & Policies**: Granular permissions for Lambda functions to access S3, DynamoDB, RDS (via Secrets Manager), Secrets Manager, and CloudWatch Logs. #### 3. **Storage (`modules/storage`, `modules/database`, `environments/dev/main.tf`)** - **S3 Buckets**: - `{project_name}-{stage}-documents`: Stores uploaded documents. S3 event notifications trigger the `document_processor` Lambda. Configured with CORS and lifecycle rules. - `{project_name}-{stage}-lambda-code`: Stores Lambda function deployment packages (ZIP files). - `{project_name}-terraform-state`: Central S3 bucket for storing Terraform state files (versioning enabled). - **DynamoDB**: - `{project_name}-{stage}-metadata`: Stores metadata related to documents (e.g., status, S3 key, user ID). Used by `upload_handler` and `document_processor`. Features Global Secondary Indexes (GSIs) on `user_id` and `document_id`, and Point-in-Time Recovery (PITR). - `{project_name}-{stage}-terraform-state-lock`: DynamoDB table for Terraform state locking, ensuring safe concurrent operations. - **PostgreSQL RDS with `pgvector` (`modules/database`)**: - Managed PostgreSQL database instance. - Utilizes the `pgvector` extension for efficient storage and similarity search of text embeddings. - Stores structured document information in a `documents` table and text chunks with their corresponding vector embeddings in a `chunks` table. - Database credentials are securely managed by AWS Secrets Manager. #### 4. **API & Authentication (`modules/api`, `modules/auth`)** - **API Gateway (REST API)**: - Provides public HTTP(S) endpoints for backend Lambda functions. - Routes include `/upload`, `/query`, and `/auth`. - Configured with CORS for frontend integration. - Amazon API Gateway has a default timeout of 30 seconds. However, GenAI use cases may require longer processing times. To support this, you can request an increased timeout via the AWS support form. After logging into your AWS account, use the following URL to access the form. In our case, we’ve configured the timeout to 150,000 milliseconds (2.5 minutes). Select United States (N. Virginia) as the region since it's set as the default in terraform.tfvars. If you're using a different region, choose the appropriate one accordingly. Keep all other settings unchanged. https://us-east-1.console.aws.amazon.com/servicequotas/home/template/add - **Cognito User Pools**: - Manages user identities, including registration, sign-in, email verification, and password reset functionalities. - Defines password policies and user attributes. - Issues JWT (JSON Web Tokens) upon successful authentication. - Includes an App Client configured for the frontend application. - **JWT-based API Authorization**: - API Gateway utilizes a Cognito JWT authorizer to protect the `/upload` and `/query` endpoints, ensuring only authenticated users can access them. - The `/auth` endpoint is public to allow user registration and login. - **Secrets Management (`modules/compute`, `modules/database`)**: - **AWS Secrets Manager**: Used to securely store and manage sensitive information: - `{project_name}-{stage}-gemini-api-key`: Stores the Google Gemini API Key used by `document_processor` and `query_processor`. - `{project_name}-{stage}-db-credentials`: Stores the master credentials for the PostgreSQL RDS instance, automatically rotated or managed by Terraform. #### 5. **Monitoring & Alerts (`modules/monitoring`)** - **CloudWatch Logs**: Centralized logging for API Gateway requests and all Lambda function executions. Log groups are configured with retention policies. - **CloudWatch Alarms**: Monitors key metrics for Lambda functions (e.g., `Errors` for `document_processor`, `query_processor`). - **SNS Topic (`{project_name}-{stage}-alerts`)**: - Acts as a notification channel. - CloudWatch Alarms publish messages to this topic when an alarm state is reached. - Can be configured with subscriptions (e.g., email) to notify administrators of issues. --- ## βš™οΈ Build and Deployment ### πŸ› οΈ Prerequisites - βœ… **Python**: `3.11+` (For Streamlit UI). - βœ… **AWS Cloud Account**: You’ll need an AWS account to build and deploy this end-to-end application (excluding the streamlit UI, which can runs locally on your system). πŸ‘‰ [AWS Free Account](https://aws.amazon.com/free/?trk=ac458a05-be8b-40e0-8d01-a185a5530151&sc_channel=ps&ef_id=Cj0KCQjw0LDBBhCnARIsAMpYlAqh7SSXOWwVg9QY_kYOXcrrp4IH9FEnyCVj77ulQ2Bok0aCDHNxpTwaAl6jEALw_wcB:G:s&s_kwcid=AL!4422!3!733868005590!e!!g!!aws%20console!22269308134!170505082450&gad_campaignid=22269308134&gbraid=0AAAAADjHtp9YddvDzIUJZUVVM05tVvxI1&gclid=Cj0KCQjw0LDBBhCnARIsAMpYlAqh7SSXOWwVg9QY_kYOXcrrp4IH9FEnyCVj77ulQ2Bok0aCDHNxpTwaAl6jEALw_wcB&all-free-tier.sort-by=item.additionalFields.SortRank&all-free-tier.sort-order=asc&awsf.Free%20Tier%20Types=*all&awsf.Free%20Tier%20Categories=*all) - βœ… **GitHub Account**: For forking the repository and using [GitHub](https://github.com/) Actions. - βœ… **Git installed on Local Machine**: Use [Git Bash](https://git-scm.com/downloads) or any preferred Git client to manage your repository. - βœ… **Google API Key**: For accessing Google's free-tier Gemini Pro and Gemini Embedding models. πŸ‘‰ [Get your API key from Google AI Studio](https://aistudio.google.com/apikey) - βœ… Free SonarCloud Account for Code Quality Checks (Optional) Sign up at [SonarCloud](https://sonarcloud.io/login?return_to=%2Fexplore%2Fprojects) to enable automated quality gates and static analysis for your codebase. --- ### 🌍 Environment Management The repository supports multiple deployment environments, typically: * `dev`: For development and testing. * `staging`: For pre-production validation. * `prod`: For the live production environment. Configuration for each environment (Terraform variables, backend configuration) is managed within its respective subfolder under the `environments/` directory (e.g., `environments/dev/`, `environments/staging/`). --- ### πŸš€ Build and Deployment #### πŸ” Set Up GitHub Repo for Build & Deployment 1. Fork the Repository πŸ‘‰ https://github.com/genieincodebottle/rag-app-on-aws 2. Clone to Your Local Machine: ``` git clone https://github.com//rag-app-on-aws.git ``` 3. Customize Project Configuration: Update the following fields in environments//terraform.tfvars: * `project_name = ""` – to avoid global resource name conflicts (e.g., S3 buckets). * `github_repo = "/rag-app-on-aws"` – for CI/CD pipeline setup. * `alert_email = ""` – for receiving deployment alerts. #### πŸ” Setting Up GitHub Secrets 1. **AWS Access Keys**: … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source β€” we do not rehost the code. - **Author:** [genieincodebottle](https://github.com/genieincodebottle) - **Source:** [genieincodebottle/rag-app-on-aws](https://github.com/genieincodebottle/rag-app-on-aws) - **License:** MIT - **Homepage:** https://aimlcompanion.ai/ Install and usage instructions live in the source repository linked above. ## Pricing - **Free** β€” Free ## Security capabilities Automated source analysis of v0.1.0 β€” what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** yes - **Dynamic code execution:** no *"Yes" means the capability is present in the source β€” more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** β€” security scan: passed β€” Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-genieincodebottle-rag-app-on-aws - Seller: https://agentstack.voostack.com/s/genieincodebottle - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack β€” the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.