# Sre Agent > SRE Agent — An AI-powered MCP server for production incident triage. Takes natural-language symptom reports, plans structured investigations using Gemini, executes parallel workers (logs, metrics, deploys, runbooks), synthesizes root-cause reports, and proposes remediation patches with human approval gates. - **Type:** MCP server - **Install:** `agentstack add mcp-benmoggee-sre-agent` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [benmoggee](https://agentstack.voostack.com/s/benmoggee) - **Installs:** 0 - **Category:** [Data & Analytics](https://agentstack.voostack.com/c/data-and-analytics) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [benmoggee](https://github.com/benmoggee) - **Source:** https://github.com/benmoggee/sre-agent ## Install ```sh agentstack add mcp-benmoggee-sre-agent ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # SRE Agent MCP Server ## Table of Contents - [The Problem](#the-problem) - [What SRE Agent Does](#what-sre-agent-does) - [Demo](#demo) - [Architecture & System Design](#architecture--system-design) - [Project Structure](#project-structure) - [Workers](#workers) - [MCP Interface](#mcp-interface) - [Dataset Scenarios](#dataset-scenarios) - [Setup Instructions](#setup-instructions) - [Environment Variables](#environment-variables) - [Running the Server](#running-the-server) - [Connecting from an MCP Client](#connecting-from-an-mcp-client) - [Example Queries](#example-queries) - [Testing](#testing) - [Troubleshooting](#troubleshooting) - [Mandatory Features Implemented](#mandatory-features-implemented) - [Custom Features Implemented](#custom-features-implemented) - [Contributing](#contributing) - [License](#license) --- ## The Problem Production incident response is one of the most high-pressure, time-sensitive activities in software engineering. When a service goes down at 3 AM, an on-call Site Reliability Engineer (SRE) is paged and must answer a cascade of questions under extreme time pressure: - **What is actually broken?** Is it a single endpoint, an entire service, or a cascading failure across multiple systems? - **What changed?** Was there a recent deployment, a configuration change, or an infrastructure event that correlates with the start of the incident? - **Where is the evidence?** Logs, metrics, and traces are scattered across different observability platforms. The engineer must query each one individually, mentally correlate timestamps, and piece together a timeline. - **What does the runbook say?** Most teams maintain runbooks for known failure modes, but finding the right runbook and following its steps while simultaneously investigating is cognitively demanding. - **What is the fix?** Once the root cause is identified, the engineer must decide on remediation: rollback a deployment, scale infrastructure, toggle a feature flag, or apply a hotfix. Each option carries risk. This process is slow, error-prone, and mentally exhausting. Studies show that Mean Time to Resolution (MTTR) for production incidents averages 1-4 hours across the industry, with a significant portion of that time spent on the investigation phase rather than the actual fix. During an outage, every minute costs money: lost revenue, SLA violations, customer churn, and engineering productivity drain. The core challenge is that **incident triage is fundamentally a multi-step reasoning task** that requires gathering evidence from multiple sources, forming hypotheses, testing them against data, and arriving at a root cause. Today, this reasoning happens entirely in the engineer's head, with no structured framework to guide the investigation or prevent cognitive shortcuts that lead to wrong conclusions. ### Why Existing Tools Fall Short Current incident response tooling addresses individual pieces of the puzzle but not the investigation workflow itself: - **Observability platforms** (Datadog, Grafana, New Relic) are excellent at storing and visualizing data, but they don't reason about what the data means or guide investigation. - **Alerting systems** (PagerDuty, OpsGenie) notify the right person, but provide no investigation support beyond the initial alert context. - **Runbook tools** (Confluence, Notion) store institutional knowledge, but there's no automated way to match symptoms to the right runbook or verify that its steps are being followed. - **ChatOps bots** (custom Slack bots) can query individual data sources, but lack the ability to plan a structured investigation or synthesize findings across sources. What's missing is an **intelligent orchestration layer** that can take a natural-language description of symptoms, plan a structured investigation, execute it by querying the right tools in the right order, and synthesize the results into an actionable incident report with remediation proposals. ## What SRE Agent Does SRE Agent is a domain-specific MCP server that acts as an **AI-powered first responder** for production incidents. It bridges the gap between raw observability data and actionable incident response by automating the investigation workflow that SREs currently perform manually. When an on-call engineer reports symptoms (e.g. *"Prod checkout is spiking 500s since 14:05 UTC"*), the server: 1. **Plans the investigation** — An LLM-powered orchestrator analyzes the symptoms, generates ranked hypotheses (e.g. "bad deployment", "dependency failure", "resource exhaustion"), and produces a structured triage plan with phased tasks and dependency ordering. 2. **Executes the investigation in parallel** — Specialized workers query logs, metrics, deployment records, and runbooks simultaneously. Same-phase tasks run in parallel for speed; cross-phase dependencies are enforced automatically. 3. **Synthesizes findings into an incident report** — Worker outputs are aggregated and analyzed to produce a root-cause assessment, evidence summary, severity classification, and recommended next actions. 4. **Proposes remediation with human approval** — The system generates patch artifacts (e.g. rollback diffs, runbook updates) but never applies them automatically. A human approval gate ensures the engineer reviews and explicitly approves any changes before they take effect. The result: an engineer who would normally spend 30-60 minutes querying dashboards, reading logs, and cross-referencing deployment history can get a structured incident report with root-cause analysis and remediation options in under a minute. ### Use Cases - **On-call triage acceleration** — Reduce MTTR by automating evidence gathering and correlation during production incidents. - **Incident postmortem preparation** — Generate structured timelines and evidence summaries that feed directly into postmortem documents. - **Runbook validation** — Automatically match incident symptoms to relevant runbooks and verify that remediation steps are applicable. - **Junior SRE training** — The structured triage plan serves as an educational tool, showing less experienced engineers how to systematically investigate an outage. - **AI-assisted pair debugging** — Use the MCP client interactively to explore an incident step by step, with the agent providing context and suggestions at each stage. --- ## Demo [demo_sre_agent.mp4](docs/demo/demo_sre_agent.mp4) --- ## Architecture & System Design The system follows Clean Architecture with an explicit composition root, ensuring that business logic is decoupled from infrastructure concerns and the MCP protocol boundary. ### Layering | Layer | Path | Responsibility | |-------|------|----------------| | **Entities** | `mcp_server/src/entities/` | Pure domain models (Pydantic). No external dependencies. | | **Use Cases** | `mcp_server/src/app/` | Orchestration, phase execution, synthesis, worker dispatch. Depends only on port interfaces (`ports.py`). | | **Infrastructure** | `mcp_server/src/infrastructure/` | Adapter implementations: Gemini LLM gateway, SRE tool gateway, approval store. | | **MCP Routers** | `mcp_server/src/routers/` | Protocol boundary. Registers tools, resources, and prompts with FastMCP. | | **Composition Root** | `mcp_server/src/workflows/main_workflow.py` | Wires infrastructure adapters into use cases and drives the end-to-end flow. | ### Ports and Adapters Dependency injection interfaces are defined in `mcp_server/src/app/ports.py`: - **`LanguageModelPort`** — `generate_json()` and `generate_text()` for LLM calls - **`SreToolGatewayPort`** — logs, metrics, deploys, runbooks, and patch operations - **`ApprovalStorePort`** — create, approve, and apply patch proposals Concrete adapters in `mcp_server/src/infrastructure/gateways.py`: - `GeminiLanguageModelGateway` — Google Gemini with structured output - `SreGuardianToolGateway` — backed by local `mcp_tools/sre_guardian` - `PatchApprovalStoreGateway` — in-memory approval store ### Workflow Phases 1. **Start** — Accepts user query and initializes progress tracking. 2. **Orchestrator** — Builds an `IncidentTriagePlan` with hypotheses, tasks, dependencies, and phase numbers. Uses Gemini structured output or deterministic fallback. 3. **Worker Execution** — Groups tasks by phase. Runs same-phase tasks in parallel via `asyncio.gather()`. Enforces `depends_on` gates before downstream tasks. 4. **Synthesis** — Aggregates worker outputs into a structured incident report with root-cause assessment, evidence, and next actions. Creates a patch proposal. 5. **Approval Gate** — Patch proposals remain pending until explicit human approval. If approved, the patch is applied; otherwise it stays unapplied. 6. **Completion** — Returns final structured payload including plan, per-task results, synthesis output, and approval state. --- ## Project Structure ```text . ├── mcp_server/ │ └── src/ │ ├── server.py # FastMCP entry point │ ├── routers/ # MCP tool/resource/prompt registration │ │ ├── tools.py │ │ ├── resources.py │ │ └── prompts.py │ ├── app/ # Use-case logic │ │ ├── ports.py # Dependency injection interfaces │ │ ├── orchestrator.py # Query -> IncidentTriagePlan │ │ ├── phase_runner.py # Phased parallel task execution │ │ ├── task_runner.py # Single task dispatch │ │ ├── synthesizer.py # Findings -> report + patch │ │ └── worker_registry.py # Tool name -> worker mapping │ ├── infrastructure/ │ │ └── gateways.py # Gemini, SRE tool, approval adapters │ ├── entities/ # Pydantic domain models │ │ ├── incident_triage_plan.py │ │ ├── tasks.py │ │ ├── workers.py │ │ ├── hypothesis.py │ │ ├── severity.py │ │ └── ... │ ├── nodes/ # Concrete worker implementations │ │ ├── base.py # BaseWorker abstract class │ │ ├── logs_query_worker.py │ │ ├── metrics_query_worker.py │ │ ├── deploy_list_worker.py │ │ ├── incident_context_worker.py │ │ ├── runbooks_search_worker.py │ │ ├── runbook_get_worker.py │ │ └── patch_generate_worker.py │ ├── tools/ # Tool implementation layer │ ├── workflows/ │ │ └── main_workflow.py # Composition root │ ├── mcp_tools/ │ │ └── sre_guardian.py # Local fallback tool backends │ ├── prompts/ # LLM prompt templates │ ├── resources/ # MCP resource implementations │ ├── config/ │ │ └── settings.py # Environment-based config (Pydantic) │ ├── models/ │ │ └── get_model.py # Gemini client initialization │ └── utils/ │ ├── approval_store.py # In-memory patch proposals │ ├── opik_utils.py # Observability integration │ └── ... ├── mcp_client/ │ └── src/ │ ├── client.py # Interactive MCP client │ ├── settings.py # Client configuration │ └── utils/ # Agent loop, LLM, parsing, commands ├── data/scenarios/ # Sample incident data │ ├── bad_deploy/ │ ├── dependency_latency/ │ └── saturation_db_pool/ ├── docs/ ├── pyproject.toml ├── .python-version ├── .env.sample └── uv.lock ``` --- ## Workers All workers inherit from `BaseWorker` and implement `async run(task, context_id) -> WorkerResult`. | Worker | Tool | Purpose | |--------|------|---------| | `LogsQueryWorker` | `logs_query` | Query structured logs by service, level, and time range | | `MetricsQueryWorker` | `metrics_query` | Fetch time-series metrics (e.g. `http_5xx_rate`) | | `DeploysListWorker` | `deploys_list` | List recent deployments in the incident window | | `IncidentContextWorker` | `incident_get_context` | Retrieve incident context and environment info | | `RunbooksSearchWorker` | `runbooks_search` | Search runbooks by symptom or keyword | | `RunbookGetWorker` | `runbooks_get` | Fetch full runbook content by ID | | `PatchGenerateWorker` | `patch_generate` | Generate rollback/remediation diffs | --- ## MCP Interface ### Tools Registered in `mcp_server/src/routers/tools.py`: - **Context management:** `incident_start_context`, `incident_get_context` - **Observability:** `logs_query`, `metrics_query` - **Deployment:** `deploys_list` - **Runbooks:** `runbooks_search`, `runbooks_get` - **Patching:** `patch_generate` - **Approval:** `request_patch_approval`, `set_patch_approval`, `apply_approved_patch`, `get_patch_approval_status` - **Workflow:** `process_user_query_workflow` (main orchestrator entry point) ### Resources - `system://status` — System health (CPU, uptime) - `system://memory` — Memory usage stats ### Prompts - `sre_triage_execution_prompt` — Full SRE instructions for incident triage. This prompt ties together the investigation tools into a guided agentic workflow and includes steps where user feedback and confirmation are required (e.g. reviewing the triage plan, approving patch proposals before application). --- ## Dataset Scenarios Sample incident datasets in `data/scenarios/` for local fallback and demonstration: | Scenario | Description | |----------|-------------| | `bad_deploy` | A code deployment introduces a connection pool exhaustion bug, causing 500 errors | | `dependency_latency` | A downstream service dependency starts timing out, cascading failures upstream | | `saturation_db_pool` | Database connection pool saturation under increased load | Each scenario includes `logs.jsonl`, `metrics.json`, `deploys.json`, and `runbooks.md`. These are consumed by `mcp_server/src/mcp_tools/sre_guardian.py` when local fallback is enabled. --- ## Setup Instructions ### Prerequisites - Python 3.14+ - [uv](https://docs.astral.sh/uv/) (fast Python package manager) ### Installation 1. Clone the repository: ```bash git clone https://github.com/your-username/sre-agent.git cd sre-agent ``` 2. Install dependencies: ```bash uv sync ``` 3. Create your environment file from the sample: ```bash cp .env.sample .env ``` 4. Fill in `.env` with your real keys (see [Environment Variables](#environment-variables) below). --- ## Environment Variables A sample file is provided at `.env.sample`. Copy it to `.env` and replace the placeholder values. | Variable | Required | Default | Description | |----------|----------|---------|-------------| | `GOOGLE_API_KEY` | **Yes** | — | Gemini API key. Get it from [Google AI Studio](https://aistudio.google.com/app/apikey). | | `MODEL_ID` | No | `gemini-2.5-flash` | Gemini model ID to use for orchestration and synthesis. | | `ORCHESTRATOR_ENABLE_OFFLINE_FALLBACK` | No | `true` | Use deterministic task plan when LLM is unavailable. | | `MCP_ENABLE_LOCAL_FALLBACK` | No | `true` | Use local scenario data when MCP backend is unavailable. | | `SRE_SAMPLE_SCENARIO` | No | `bad_deploy` | Which scenario to load (`bad_deploy`, `dependency_latency`, `saturation_db_pool`). | | `LOG_LEVEL` | No | `20` | Logging level (10=DEBUG, 20=INFO, 30=WARNING). | | `LOG_LEVEL_DEPENDENCIES` | No | `30` | Logging level for third-party libraries. | | `OPIK_API_KEY` | No | — | [Opik](https://www.comet.com/opik) API key for observability. | | `OPIK_WORKSPACE` | No | — | Opik workspace name. | | `OPIK_PROJECT_NAME` | No | `sre_guardian` | Opik project name. | | `OPENAI_API_KEY` | No | — | OpenAI API key (reserved for future provider support). | --- ## Running the Server ### MCP Server (stdio) ```bash uv --directory mcp_server run -m src.server --transport stdio ``` ### MCP Server (streama … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [benmoggee](https://github.com/benmoggee) - **Source:** [benmoggee/sre-agent](https://github.com/benmoggee/sre-agent) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** yes - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-benmoggee-sre-agent - Seller: https://agentstack.voostack.com/s/benmoggee - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.