Rag App On Aws

   

👉 GenAI Roadmap - 2025

End-to-End RAG App with Evaluation on AWS, Integrating Web Search via Remote MCP Server

Terraform-based Infrastructure as Code (IaC) to deploy a complete AWS backend for a Retrieval-Augmented Generation (RAG) application. It integrates with Google’s free-tier Gemini Pro and Embedding models for AI powered document querying and includes a Streamlit UI with token-based authentication for interacting with the app.

👉 Related Remote MCP Server: Web Search using SerpAPI Remote MCP Server based on Streaming Http Transport protocol for Real Time Web Search. It's located within the mcp_servers/ directory of this repository.

👉 Related UI: RAG UI (Streamlit Frontend) A Streamlit-based frontend application designed to interact with the backend infrastructure deployed by this project. It's located within the rag_ui/ directory of this repository.

💰 Estimated cost: ~$3 (~₹250) to experiment without the AWS Free Tier, primarily for RDS and NAT Gateway if active.

🎥 YouTube Video: Walkthrough on setting up the application, building, deploying, and running it end-to-end 👇

[](https://www.youtube.com/watch?v=x2P4Ee6PYNg)

---

🔍 Overview

This repository contains the complete Terraform codebase for provisioning and managing the AWS infrastructure that powers a RAG application. It allows users to upload documents, which are then processed, embedded, and stored for efficient semantic search and AI-driven querying.

📌 Key features include:

• IaC with Terraform: For consistent and repeatable deployments across environments.
• Serverless Compute: AWS Lambda for backend logic (document processing, querying, uploads, authentication, DB initialization).
• Vector Storage: PostgreSQL RDS with the pgvector extension for storing and searching text embeddings.
• AI Integration: Leverages Google's Gemini Pro (for generation) and Gemini Embedding models (for text embeddings).
• Authentication: Secure user management with AWS Cognito.
• CI/CD Workflows: GitHub Actions for automated deployment, testing, and cleanup.
• Multi-Environment Support: Designed for dev, staging, and production environments.
• Comprehensive Testing: Includes unit and integration tests for backend Lambda functions.
• Streamlit UI: Includes a login page, document upload, query interface, and RAG evaluation dashboard.

---

🏗️ High Level Architecture

---

🌐 Network Flow Walkthrough (Referencing the Architecture)

🗂️ Document Processing Flow with Network Components:

1. User uploads document → API Gateway → upload_handler Lambda
2. upload_handler Lambda → S3 Gateway Endpoint → S3 Bucket
3. S3 Event → document_processor Lambda (in private subnet)
4. document_processor Lambda → NAT Gateway → Internet Gateway → Gemini API (for embeddings)
5. document_processor Lambda → RDS Security Group → PostgreSQL Database (stores chunks/vectors)

💬 Query Processing Flow with Network Components:

1. User submits query → API Gateway → query_processor Lambda (in private subnet)
2. query_processor Lambda → RDS Security Group → PostgreSQL Database (vector search)
3. query_processor Lambda → NAT Gateway → Internet Gateway → Gemini API (for answer generation)
4. query_processor Lambda → API Gateway → User (returns answer)

This network architecture ensures that sensitive operations and data are processed in a secure environment, while still allowing the necessary external communications through controlled channels.

---

🔁 GitHub Action Pipeline

---

🔁 AWS Infra Provisioning Flow Diagram

🗺️ Infra Provisioning Lifecycle Flow (Illustrates the Terraform provisioning sequence)

---

🗂️ Repository Structure

.
├── .github/workflows/       # CI/CD via GitHub Actions
│   ├── deploy.yml           # Infrastructure deployment workflow
│   └── manual_cleanup.yml   # Resource cleanup workflow
├── environments/            # Environment-specific configs (dev, staging, prod)
│   └── dev/                 # Example 'dev' environment
│       ├── main.tf          # Root Terraform file for the environment
│       ├── providers.tf     # Terraform provider configurations
│       └── variables.tf     # Environment-specific variable definitions
├── modules/                 # Reusable Terraform modules
│   ├── api/                 # API Gateway configuration
│   ├── auth/                # Cognito authentication
│   ├── compute/             # Lambda functions & IAM roles
│   ├── database/            # PostgreSQL RDS with pgvector & Secrets Manager
│   ├── monitoring/          # CloudWatch Logs, Alarms & SNS Topic
│   ├── storage/             # S3 Buckets & DynamoDB Table
│   └── vpc/                 # VPC, Subnets, NAT, Security Groups, Endpoints
├── rag_ui/                  # Streamlit UI application
│   ├── app.py               # Main Streamlit application code
│   └── README.md            # README specific to the UI
├── scripts/                 # Utility shell scripts
│   ├── cleanup.sh           # Comprehensive resource cleanup script
│   ├── import_resources.sh  # Script to import existing AWS resources into Terraform state
│   └── network-diagnostics.sh # Script for troubleshooting network connectivity (e.g., Lambda to RDS)
├── src/                     # Lambda backend source code (Python)
│   ├── auth_handler/        # Lambda for Cognito authentication operations
│   ├── db_init/             # Lambda for database schema and pgvector initialization
│   ├── document_processor/  # Lambda for processing uploaded documents
│   ├── query_processor/     # Lambda for handling user queries and RAG
│   ├── tests/               # Unit and integration tests
│   │   ├── integration/     # Integration tests for deployed services
│   │   │   └── run_integration_tests.py
│   │   ├── unit/            # Unit tests for Lambda functions
│   │   │   ├── conftest.py  # Pytest common fixtures and mocks
│   │   │   ├── test_*.py    # Individual unit test files
│   │   └── __init__.py
│   ├── upload_handler/      # Lambda for handling file uploads via API
│   └── utils/               # Shared utility code (e.g., db_connectivity_test.py)
├── sonar-project.properties # SonarQube configuration file
└── tox.ini                  # tox configuration for running tests and linters

---

🧱 Infrastructure Components

The infrastructure is modularized using Terraform modules:

1. Networking (VPC - modules/vpc)

• Custom VPC with public, private, and database subnets across multiple Availability Zones.
• Internet Gateway for public subnet access.
• NAT Gateways (configurable for single or multiple AZs) for private subnet outbound access.
• Route Tables for managing traffic flow.
• Security Groups to control access to Lambdas, RDS, and Bastion hosts.
• VPC Endpoints for S3 and DynamoDB, allowing private access from within the VPC.
• Optional VPC Flow Logs for network traffic monitoring (enabled for prod).

2. Compute (Lambda Functions - modules/compute, src/)

• All functions are Python 3.11 based.
• Authentication Handler (auth_handler): Manages user authentication lifecycle with Cognito (registration, login, email verification, password reset, token refresh).
• Document Processor (document_processor):
• Triggered by S3 uploads to the uploads/ prefix in the documents bucket.
• Downloads the uploaded file (PDF, TXT, CSV, etc.).
• Loads and chunks the document content.
• Generates text embeddings for chunks using the Gemini Embedding model.
• Stores document metadata and text chunks (with embeddings) in the PostgreSQL RDS database.
• Query Processor (query_processor):
• Handles user queries from the API.
• Generates an embedding for the user's query using the Gemini Embedding model.
• Performs a vector similarity search in PostgreSQL (using pgvector) against stored document chunks.
• Retrieves relevant chunks and prepares a context.
• Generates a final answer using the Gemini Pro model with the retrieved context.
• Optionally performs RAG evaluation (faithfulness, relevancy, context precision).
• Upload Handler (upload_handler):
• API endpoint for initiating file uploads.
• Receives file content (base64 encoded), name, and user ID.
• Uploads the raw file to a specific S3 path (uploads/{user_id}/{document_id}/{file_name}).
• Stores initial document metadata in PostgreSQL and DynamoDB.
• DB Initialization (db_init):
• A Lambda function invoked during CI/CD deployment.
• Connects to the PostgreSQL RDS instance.
• Creates necessary database tables (documents, chunks) if they don't exist.
• Enables the pgvector extension required for vector operations.
• IAM Roles & Policies: Granular permissions for Lambda functions to access S3, DynamoDB, RDS (via Secrets Manager), Secrets Manager, and CloudWatch Logs.

3. Storage (modules/storage, modules/database, environments/dev/main.tf)

• S3 Buckets:
• {project_name}-{stage}-documents: Stores uploaded documents. S3 event notifications trigger the document_processor Lambda. Configured with CORS and lifecycle rules.
• {project_name}-{stage}-lambda-code: Stores Lambda function deployment packages (ZIP files).
• {project_name}-terraform-state: Central S3 bucket for storing Terraform state files (versioning enabled).
• DynamoDB:
• {project_name}-{stage}-metadata: Stores metadata related to documents (e.g., status, S3 key, user ID). Used by upload_handler and document_processor. Features Global Secondary Indexes (GSIs) on user_id and document_id, and Point-in-Time Recovery (PITR).
• {project_name}-{stage}-terraform-state-lock: DynamoDB table for Terraform state locking, ensuring safe concurrent operations.
• PostgreSQL RDS with pgvector (modules/database):
• Managed PostgreSQL database instance.
• Utilizes the pgvector extension for efficient storage and similarity search of text embeddings.
• Stores structured document information in a documents table and text chunks with their corresponding vector embeddings in a chunks table.
• Database credentials are securely managed by AWS Secrets Manager.

4. API & Authentication (modules/api, modules/auth)

• API Gateway (REST API):
• Provides public HTTP(S) endpoints for backend Lambda functions.
• Routes include /upload, /query, and /auth.
• Configured with CORS for frontend integration.
• Amazon API Gateway has a default timeout of 30 seconds. However, GenAI use cases may require longer processing times. To support this, you can request an increased timeout via the AWS support form. After logging into your AWS account, use the following URL to access the form. In our case, we’ve configured the timeout to 150,000 milliseconds (2.5 minutes). Select United States (N. Virginia) as the region since it's set as the default in terraform.tfvars. If you're using a different region, choose the appropriate one accordingly. Keep all other settings unchanged.

https://us-east-1.console.aws.amazon.com/servicequotas/home/template/add

• Cognito User Pools:
• Manages user identities, including registration, sign-in, email verification, and password reset functionalities.
• Defines password policies and user attributes.
• Issues JWT (JSON Web Tokens) upon successful authentication.
• Includes an App Client configured for the frontend application.
• JWT-based API Authorization:
• API Gateway utilizes a Cognito JWT authorizer to protect the /upload and /query endpoints, ensuring only authenticated users can access them.
• The /auth endpoint is public to allow user registration and login.
• Secrets Management (modules/compute, modules/database):
• AWS Secrets Manager: Used to securely store and manage sensitive information:
• {project_name}-{stage}-gemini-api-key: Stores the Google Gemini API Key used by document_processor and query_processor.
• {project_name}-{stage}-db-credentials: Stores the master credentials for the PostgreSQL RDS instance, automatically rotated or managed by Terraform.

5. Monitoring & Alerts (modules/monitoring)

• CloudWatch Logs: Centralized logging for API Gateway requests and all Lambda function executions. Log groups are configured with retention policies.
• CloudWatch Alarms: Monitors key metrics for Lambda functions (e.g., Errors for document_processor, query_processor).
• SNS Topic ({project_name}-{stage}-alerts):
• Acts as a notification channel.
• CloudWatch Alarms publish messages to this topic when an alarm state is reached.
• Can be configured with subscriptions (e.g., email) to notify administrators of issues.

---

⚙️ Build and Deployment

🛠️ Prerequisites

• ✅ Python: 3.11+ (For Streamlit UI).
• ✅ AWS Cloud Account: You’ll need an AWS account to build and deploy this end-to-end application (excluding the streamlit UI, which can runs locally on your system).

👉 AWS Free Account

• ✅ GitHub Account: For forking the repository and using GitHub Actions.
• ✅ Git installed on Local Machine: Use Git Bash or any preferred Git client to manage your repository.
• ✅ Google API Key: For accessing Google's free-tier Gemini Pro and Gemini Embedding models.

👉 Get your API key from Google AI Studio

• ✅ Free SonarCloud Account for Code Quality Checks (Optional)

Sign up at SonarCloud to enable automated quality gates and static analysis for your codebase.

---

🌍 Environment Management

The repository supports multiple deployment environments, typically:

• dev: For development and testing.
• staging: For pre-production validation.
• prod: For the live production environment.

Configuration for each environment (Terraform variables, backend configuration) is managed within its respective subfolder under the environments/ directory (e.g., environments/dev/, environments/staging/).

---

🚀 Build and Deployment

🔐 Set Up GitHub Repo for Build & Deployment

1. Fork the Repository

👉 https://github.com/genieincodebottle/rag-app-on-aws

1. Clone to Your Local Machine:

`` git clone https://github.com//rag-app-on-aws.git ``

1. Customize Project Configuration:

Update the following fields in environments//terraform.tfvars:

• project_name = "" – to avoid global resource name conflicts (e.g., S3 buckets).
• github_repo = "/rag-app-on-aws" – for CI/CD pipeline setup.
• alert_email = "" – for receiving deployment alerts.

🔐 Setting Up GitHub Secrets

1. AWS Access Keys:

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: genieincodebottle
• Source: genieincodebottle/rag-app-on-aws
• License: MIT
• Homepage: https://aimlcompanion.ai/

Install and usage instructions live in the source repository linked above.