Sre Agent

SRE Agent MCP Server

Table of Contents

• [The Problem](#the-problem)
• [What SRE Agent Does](#what-sre-agent-does)
• [Demo](#demo)
• [Architecture & System Design](#architecture--system-design)
• [Project Structure](#project-structure)
• [Workers](#workers)
• [MCP Interface](#mcp-interface)
• [Dataset Scenarios](#dataset-scenarios)
• [Setup Instructions](#setup-instructions)
• [Environment Variables](#environment-variables)
• [Running the Server](#running-the-server)
• [Connecting from an MCP Client](#connecting-from-an-mcp-client)
• [Example Queries](#example-queries)
• [Testing](#testing)
• [Troubleshooting](#troubleshooting)
• [Mandatory Features Implemented](#mandatory-features-implemented)
• [Custom Features Implemented](#custom-features-implemented)
• [Contributing](#contributing)
• [License](#license)

---

The Problem

Production incident response is one of the most high-pressure, time-sensitive activities in software engineering. When a service goes down at 3 AM, an on-call Site Reliability Engineer (SRE) is paged and must answer a cascade of questions under extreme time pressure:

• What is actually broken? Is it a single endpoint, an entire service, or a cascading failure across multiple systems?
• What changed? Was there a recent deployment, a configuration change, or an infrastructure event that correlates with the start of the incident?
• Where is the evidence? Logs, metrics, and traces are scattered across different observability platforms. The engineer must query each one individually, mentally correlate timestamps, and piece together a timeline.
• What does the runbook say? Most teams maintain runbooks for known failure modes, but finding the right runbook and following its steps while simultaneously investigating is cognitively demanding.
• What is the fix? Once the root cause is identified, the engineer must decide on remediation: rollback a deployment, scale infrastructure, toggle a feature flag, or apply a hotfix. Each option carries risk.

This process is slow, error-prone, and mentally exhausting. Studies show that Mean Time to Resolution (MTTR) for production incidents averages 1-4 hours across the industry, with a significant portion of that time spent on the investigation phase rather than the actual fix. During an outage, every minute costs money: lost revenue, SLA violations, customer churn, and engineering productivity drain.

The core challenge is that incident triage is fundamentally a multi-step reasoning task that requires gathering evidence from multiple sources, forming hypotheses, testing them against data, and arriving at a root cause. Today, this reasoning happens entirely in the engineer's head, with no structured framework to guide the investigation or prevent cognitive shortcuts that lead to wrong conclusions.

Why Existing Tools Fall Short

Current incident response tooling addresses individual pieces of the puzzle but not the investigation workflow itself:

• Observability platforms (Datadog, Grafana, New Relic) are excellent at storing and visualizing data, but they don't reason about what the data means or guide investigation.
• Alerting systems (PagerDuty, OpsGenie) notify the right person, but provide no investigation support beyond the initial alert context.
• Runbook tools (Confluence, Notion) store institutional knowledge, but there's no automated way to match symptoms to the right runbook or verify that its steps are being followed.
• ChatOps bots (custom Slack bots) can query individual data sources, but lack the ability to plan a structured investigation or synthesize findings across sources.

What's missing is an intelligent orchestration layer that can take a natural-language description of symptoms, plan a structured investigation, execute it by querying the right tools in the right order, and synthesize the results into an actionable incident report with remediation proposals.

What SRE Agent Does

SRE Agent is a domain-specific MCP server that acts as an AI-powered first responder for production incidents. It bridges the gap between raw observability data and actionable incident response by automating the investigation workflow that SREs currently perform manually.

When an on-call engineer reports symptoms (e.g. "Prod checkout is spiking 500s since 14:05 UTC"), the server:

1. Plans the investigation — An LLM-powered orchestrator analyzes the symptoms, generates ranked hypotheses (e.g. "bad deployment", "dependency failure", "resource exhaustion"), and produces a structured triage plan with phased tasks and dependency ordering.

1. Executes the investigation in parallel — Specialized workers query logs, metrics, deployment records, and runbooks simultaneously. Same-phase tasks run in parallel for speed; cross-phase dependencies are enforced automatically.

1. Synthesizes findings into an incident report — Worker outputs are aggregated and analyzed to produce a root-cause assessment, evidence summary, severity classification, and recommended next actions.

1. Proposes remediation with human approval — The system generates patch artifacts (e.g. rollback diffs, runbook updates) but never applies them automatically. A human approval gate ensures the engineer reviews and explicitly approves any changes before they take effect.

The result: an engineer who would normally spend 30-60 minutes querying dashboards, reading logs, and cross-referencing deployment history can get a structured incident report with root-cause analysis and remediation options in under a minute.

Use Cases

• On-call triage acceleration — Reduce MTTR by automating evidence gathering and correlation during production incidents.
• Incident postmortem preparation — Generate structured timelines and evidence summaries that feed directly into postmortem documents.
• Runbook validation — Automatically match incident symptoms to relevant runbooks and verify that remediation steps are applicable.
• Junior SRE training — The structured triage plan serves as an educational tool, showing less experienced engineers how to systematically investigate an outage.
• AI-assisted pair debugging — Use the MCP client interactively to explore an incident step by step, with the agent providing context and suggestions at each stage.

---

Demo

[demosreagent.mp4](docs/demo/demosreagent.mp4)

---

Architecture & System Design

The system follows Clean Architecture with an explicit composition root, ensuring that business logic is decoupled from infrastructure concerns and the MCP protocol boundary.

Layering

| Layer | Path | Responsibility | |-------|------|----------------| | Entities | mcp_server/src/entities/ | Pure domain models (Pydantic). No external dependencies. | | Use Cases | mcp_server/src/app/ | Orchestration, phase execution, synthesis, worker dispatch. Depends only on port interfaces (ports.py). | | Infrastructure | mcp_server/src/infrastructure/ | Adapter implementations: Gemini LLM gateway, SRE tool gateway, approval store. | | MCP Routers | mcp_server/src/routers/ | Protocol boundary. Registers tools, resources, and prompts with FastMCP. | | Composition Root | mcp_server/src/workflows/main_workflow.py | Wires infrastructure adapters into use cases and drives the end-to-end flow. |

Ports and Adapters

Dependency injection interfaces are defined in mcp_server/src/app/ports.py:

• LanguageModelPort — generate_json() and generate_text() for LLM calls
• SreToolGatewayPort — logs, metrics, deploys, runbooks, and patch operations
• ApprovalStorePort — create, approve, and apply patch proposals

Concrete adapters in mcp_server/src/infrastructure/gateways.py:

• GeminiLanguageModelGateway — Google Gemini with structured output
• SreGuardianToolGateway — backed by local mcp_tools/sre_guardian
• PatchApprovalStoreGateway — in-memory approval store

Workflow Phases

1. Start — Accepts user query and initializes progress tracking.
2. Orchestrator — Builds an IncidentTriagePlan with hypotheses, tasks, dependencies, and phase numbers. Uses Gemini structured output or deterministic fallback.
3. Worker Execution — Groups tasks by phase. Runs same-phase tasks in parallel via asyncio.gather(). Enforces depends_on gates before downstream tasks.
4. Synthesis — Aggregates worker outputs into a structured incident report with root-cause assessment, evidence, and next actions. Creates a patch proposal.
5. Approval Gate — Patch proposals remain pending until explicit human approval. If approved, the patch is applied; otherwise it stays unapplied.
6. Completion — Returns final structured payload including plan, per-task results, synthesis output, and approval state.

---

Project Structure

.
├── mcp_server/
│   └── src/
│       ├── server.py                  # FastMCP entry point
│       ├── routers/                   # MCP tool/resource/prompt registration
│       │   ├── tools.py
│       │   ├── resources.py
│       │   └── prompts.py
│       ├── app/                       # Use-case logic
│       │   ├── ports.py               # Dependency injection interfaces
│       │   ├── orchestrator.py        # Query -> IncidentTriagePlan
│       │   ├── phase_runner.py        # Phased parallel task execution
│       │   ├── task_runner.py         # Single task dispatch
│       │   ├── synthesizer.py         # Findings -> report + patch
│       │   └── worker_registry.py     # Tool name -> worker mapping
│       ├── infrastructure/
│       │   └── gateways.py            # Gemini, SRE tool, approval adapters
│       ├── entities/                  # Pydantic domain models
│       │   ├── incident_triage_plan.py
│       │   ├── tasks.py
│       │   ├── workers.py
│       │   ├── hypothesis.py
│       │   ├── severity.py
│       │   └── ...
│       ├── nodes/                     # Concrete worker implementations
│       │   ├── base.py                # BaseWorker abstract class
│       │   ├── logs_query_worker.py
│       │   ├── metrics_query_worker.py
│       │   ├── deploy_list_worker.py
│       │   ├── incident_context_worker.py
│       │   ├── runbooks_search_worker.py
│       │   ├── runbook_get_worker.py
│       │   └── patch_generate_worker.py
│       ├── tools/                     # Tool implementation layer
│       ├── workflows/
│       │   └── main_workflow.py       # Composition root
│       ├── mcp_tools/
│       │   └── sre_guardian.py        # Local fallback tool backends
│       ├── prompts/                   # LLM prompt templates
│       ├── resources/                 # MCP resource implementations
│       ├── config/
│       │   └── settings.py            # Environment-based config (Pydantic)
│       ├── models/
│       │   └── get_model.py           # Gemini client initialization
│       └── utils/
│           ├── approval_store.py      # In-memory patch proposals
│           ├── opik_utils.py          # Observability integration
│           └── ...
├── mcp_client/
│   └── src/
│       ├── client.py                  # Interactive MCP client
│       ├── settings.py                # Client configuration
│       └── utils/                     # Agent loop, LLM, parsing, commands
├── data/scenarios/                    # Sample incident data
│   ├── bad_deploy/
│   ├── dependency_latency/
│   └── saturation_db_pool/
├── docs/
├── pyproject.toml
├── .python-version
├── .env.sample
└── uv.lock

---

Workers

All workers inherit from BaseWorker and implement async run(task, context_id) -> WorkerResult.

| Worker | Tool | Purpose | |--------|------|---------| | LogsQueryWorker | logs_query | Query structured logs by service, level, and time range | | MetricsQueryWorker | metrics_query | Fetch time-series metrics (e.g. http_5xx_rate) | | DeploysListWorker | deploys_list | List recent deployments in the incident window | | IncidentContextWorker | incident_get_context | Retrieve incident context and environment info | | RunbooksSearchWorker | runbooks_search | Search runbooks by symptom or keyword | | RunbookGetWorker | runbooks_get | Fetch full runbook content by ID | | PatchGenerateWorker | patch_generate | Generate rollback/remediation diffs |

---

MCP Interface

Tools

Registered in mcp_server/src/routers/tools.py:

• Context management: incident_start_context, incident_get_context
• Observability: logs_query, metrics_query
• Deployment: deploys_list
• Runbooks: runbooks_search, runbooks_get
• Patching: patch_generate
• Approval: request_patch_approval, set_patch_approval, apply_approved_patch, get_patch_approval_status
• Workflow: process_user_query_workflow (main orchestrator entry point)

Resources

• system://status — System health (CPU, uptime)
• system://memory — Memory usage stats

Prompts

• sre_triage_execution_prompt — Full SRE instructions for incident triage. This prompt ties together the investigation tools into a guided agentic workflow and includes steps where user feedback and confirmation are required (e.g. reviewing the triage plan, approving patch proposals before application).

---

Dataset Scenarios

Sample incident datasets in data/scenarios/ for local fallback and demonstration:

| Scenario | Description | |----------|-------------| | bad_deploy | A code deployment introduces a connection pool exhaustion bug, causing 500 errors | | dependency_latency | A downstream service dependency starts timing out, cascading failures upstream | | saturation_db_pool | Database connection pool saturation under increased load |

Each scenario includes logs.jsonl, metrics.json, deploys.json, and runbooks.md. These are consumed by mcp_server/src/mcp_tools/sre_guardian.py when local fallback is enabled.

---

Setup Instructions

Prerequisites

• Python 3.14+
• uv (fast Python package manager)

Installation

1. Clone the repository:

git clone https://github.com/your-username/sre-agent.git
cd sre-agent

1. Install dependencies:

uv sync

1. Create your environment file from the sample:

cp .env.sample .env

1. Fill in .env with your real keys (see [Environment Variables](#environment-variables) below).

---

Environment Variables

A sample file is provided at .env.sample. Copy it to .env and replace the placeholder values.

| Variable | Required | Default | Description | |----------|----------|---------|-------------| | GOOGLE_API_KEY | Yes | — | Gemini API key. Get it from Google AI Studio. | | MODEL_ID | No | gemini-2.5-flash | Gemini model ID to use for orchestration and synthesis. | | ORCHESTRATOR_ENABLE_OFFLINE_FALLBACK | No | true | Use deterministic task plan when LLM is unavailable. | | MCP_ENABLE_LOCAL_FALLBACK | No | true | Use local scenario data when MCP backend is unavailable. | | SRE_SAMPLE_SCENARIO | No | bad_deploy | Which scenario to load (bad_deploy, dependency_latency, saturation_db_pool). | | LOG_LEVEL | No | 20 | Logging level (10=DEBUG, 20=INFO, 30=WARNING). | | LOG_LEVEL_DEPENDENCIES | No | 30 | Logging level for third-party libraries. | | OPIK_API_KEY | No | — | Opik API key for observability. | | OPIK_WORKSPACE | No | — | Opik workspace name. | | OPIK_PROJECT_NAME | No | sre_guardian | Opik project name. | | OPENAI_API_KEY | No | — | OpenAI API key (reserved for future provider support). |

---

Running the Server

MCP Server (stdio)

uv --directory mcp_server run -m src.server --transport stdio

MCP Server (streama

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: benmoggee
• Source: benmoggee/sre-agent
• License: MIT

Install and usage instructions live in the source repository linked above.