Ai Infra Control Plane

AI Infrastructure Control Plane

[](https://github.com/justrunme/ai-infra-control-plane/actions/workflows/ci.yml) [](https://github.com/justrunme/ai-infra-control-plane/actions/workflows/release.yml)

> AI Infrastructure OS — open-source operating layer for private AI platforms: policy, cost, capacity, observability, and fleet operations on Kubernetes.

[](docs/videos/hero-overview.mp4)

This repository is the Control Plane of the AI Infrastructure OS. The reference Execution Plane is AI Runtime Platform.

Read the [product roadmap](docs/product-roadmap.md) and [portfolio overview](docs/portfolio-overview.md) for the full platform map.

One-command platform demo

Bring up Control Plane + Execution Plane + Ollama and verify governance enforcement:

make platform-demo              # laptop demo
make platform-demo-verify

make platform-demo-production   # + Redis quota + Prometheus inputs
make platform-demo-production-verify

make platform-demo-enterprise   # + Keycloak OIDC (reference architecture)
make platform-demo-enterprise-verify

See [demo/platform/README.md](demo/platform/README.md) for URLs and manual curls.

This project is intentionally scoped as an AI infrastructure platform, not an agent framework.

The core workflow is:

AI request
  -> tenant quota
  -> model risk registry
  -> cost decision
  -> risk score
  -> approval decision
  -> final verdict

Read the portfolio overview in docs/portfolio-overview.md, the [product roadmap](docs/product-roadmap.md), and the technical system design in docs/platform-architecture.md.

Operator Dashboard

The control API serves a live operator dashboard at / with platform status, topology health, model inventory, a governance playground, and inventory drift detection, refreshed every few seconds.

Governance Playground

> Interactive demo | Click the animated preview to watch the governance playground and drift detection walkthrough.

[](docs/videos/governance-playground.mp4)

Submit an AI platform request through the live governance pipeline without leaving the browser:

• POST /governance/evaluate — quota → registry → cost → risk → approval → final verdict
• Presets on / for low-risk dev, production external, and budget-exceeded scenarios

Inventory Drift Detection

Compare configured model inventory (MODEL_INVENTORY_PATH / Helm ConfigMap) against live Ollama and vLLM probes:

• GET /drift — desired vs actual models per backend, missing/unexpected lists
• Prometheus gauges: ai_control_inventory_in_sync, ai_control_inventory_drift

How the Projects Fit Together

This repository is the Control Plane of the [AI Infrastructure OS](docs/product-roadmap.md). The Execution Plane lives in ai-runtime-platform.

| Layer | Role in AI Infrastructure OS | Repository | | --- | --- | --- | | Execution Plane | Runs inference, routes traffic, enforces governance verdicts | justrunme/ai-runtime-platform | | Control Plane | Policy, cost, topology, drift, SLO, fleet operations | justrunme/ai-infra-control-plane |

The Execution Plane runs workloads. The Control Plane evaluates policy and the runtime enforces verdicts at the inference boundary via CONTROL_PLANE_URL.

Product Walkthroughs

Governance Decision Flow

[](docs/videos/governance-pipeline.mp4)

AI Infrastructure Digital Twin

[](docs/videos/digital-twin.mp4)

Forecast-driven Scaling

[](docs/videos/forecast-driven-scaling.mp4)

Visual Overview

Platform Overview

flowchart TB
    User["AI Consumer"]
    API["Control APIFastAPI"]

    subgraph ControlPlane["AI Infrastructure Control Plane"]
        Capacity["Capacity Planner"]
        Cost["Cost Governance"]
        Risk["Risk Scoring"]
        Approval["Approval Engine"]
        Twin["Digital Twin"]
    end

    subgraph AI["AI Workloads"]
        Ollama["Ollama"]
        VLLM["vLLM"]
        Models["Foundation Models"]
    end

    subgraph Observability["Observability"]
        OTel["OpenTelemetry"]
        Prom["Prometheus"]
        Graf["Grafana"]
        Loki["Loki"]
    end

    User --> API
    API --> Capacity
    API --> Cost
    API --> Risk
    API --> Approval
    API --> Twin
    Capacity --> Ollama
    Capacity --> VLLM
    Ollama --> Models
    VLLM --> Models
    Ollama --> OTel
    VLLM --> OTel
    OTel --> Prom
    OTel --> Loki
    Prom --> Graf
    Loki --> Graf
    Twin --> Graf

Governance Flow

flowchart LR
    Request["AI Request"]
    Cost["Cost Analysis"]
    Risk["Risk Analysis"]
    Capacity["Capacity Check"]
    Decision{"Policy Engine"}
    Allow["ALLOW"]
    Warn["WARN"]
    Block["BLOCK"]

    Request --> Cost
    Request --> Risk
    Request --> Capacity
    Cost --> Decision
    Risk --> Decision
    Capacity --> Decision
    Decision --> Allow
    Decision --> Warn
    Decision --> Block

Digital Twin

flowchart LR
    Real["Real AI Cluster"]
    Metrics["Telemetry"]
    Twin["Digital Twin Model"]
    Simulate["Scenario Simulation"]
    Decision["Capacity Decision"]

    Real --> Metrics
    Metrics --> Twin
    Twin --> Simulate
    Simulate --> Decision

Forecast-driven Scaling

flowchart TB
    Metrics["Historical Metrics"]
    TimesFM["TimesFM Forecasting"]
    Forecast["Demand Forecast"]
    ScaleUp["Scale Up"]
    ScaleDown["Scale Down"]

    Metrics --> TimesFM
    TimesFM --> Forecast
    Forecast --> ScaleUp
    Forecast --> ScaleDown

GitOps Delivery

flowchart LR
    Dev["Developer"]
    Git["Git Repository"]
    Actions["GitHub Actions"]
    Registry["Container Registry"]
    Argo["Argo CD"]
    Cluster["Kubernetes Cluster"]

    Dev --> Git
    Git --> Actions
    Actions --> Registry
    Registry --> Argo
    Argo --> Cluster

Scope

• Expose a control API for private AI backend health, latency, capacity, cost, identity, audit, and governance signals.
• Operate model, tool, agent, and intent metadata through registries and policy packs.
• Evaluate governance decisions across prompt security, tenant quota, model registry, cost, risk, approval, sovereign AI, and response evaluation stages.
• Feed live governance inputs from Redis-backed tenant quota state and Prometheus telemetry.
• Run a full reference stack with Control Plane, Execution Plane, Ollama, Redis, Prometheus, and Keycloak OIDC.
• Package the API with Docker and Helm, including production defaults and External Secrets / JWKS wiring.
• Provide GitOps, security, SLO, FinOps, fleet topology, capacity, and GPU placement reference modules.
• Keep experimental modules clearly isolated from the production deployment path.

Repository Layout

apps/
  control-api/        FastAPI service with health and model status endpoints
infra/
  helm/               Kubernetes packaging
  terraform/          Cloud bootstrap modules
    k3s-bootstrap/    Example Hetzner VM bootstrap with cloud-init and k3s
observability/
  grafana/            Dashboards and metrics notes
  loki/               Loki and Promtail logging examples
  otel-genai/         OpenTelemetry GenAI telemetry reference implementation
forecasting/
  timesfm/            Experimental capacity forecasting module
experiments/
  inference-autoscaling/ Forecast-driven inference scaling recommendations
governance/
  agents/             Agent-to-model/tool/policy bindings
  cost/               AI cost governance policy engine
  intent/             Natural-language intent to governed execution plan
  policy-packs/       Environment and team-specific policy overlays
  prompt-security/    PII, secret, and prompt-injection checks
  quota/              Tenant quota policy checks
  registry/           Signed model registry and attestation metadata
  risk/               AI request risk scoring engine
  approval/           Human approval workflow reference implementation
  pipeline/           End-to-end AI governance decision pipeline
  tools/              MCP tool registry and action allowlists
security/
  trivy/              Container and IaC scan configuration
  opa/                Kubernetes policy gates for rendered manifests
docs/
  architecture.md     System design notes
  case-study.md       Portfolio case study and demo flow
  digital-twin.md     AI infrastructure topology model
  platform-architecture.md Technical platform architecture

Local Development

cd apps/control-api
python3.12 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload

Run tests:

make venv
make test

The project targets Python 3.12 for local development and CI.

Run the portfolio demo:

make demo

The demo prints the key control API endpoints and runs the end-to-end governance pipeline from governance/pipeline/sample_requests.csv.

Control API

The control API exposes operator-facing signals for private AI infrastructure:

• GET / - live operator dashboard with governance playground and drift panel (HTML).
• GET /health - operator-facing service health.
• GET /healthz - Kubernetes-compatible health check.
• GET /models - configured model backends and status.
• GET /metrics - Prometheus-compatible text metrics.
• GET /capacity - aggregate model serving capacity.
• GET /cost - estimated hourly, daily, and monthly cost.
• GET /summary - compact status for dashboards and demos.
• GET /governance/inputs/status - Redis quota and Prometheus input status.
• POST /intent/resolve - intent-to-agent/model/tool/region execution planning.
• POST /governance/evaluate-tool - governed MCP tool call decision.
• POST /governance/evaluate-response - post-response quality, latency, and cost evaluation.

Model Inventory

The model inventory is configuration-driven. By default the API loads app/model_inventory.json shipped with the image, but you can point it at any JSON file:

export MODEL_INVENTORY_PATH=/etc/ai-control-plane/model_inventory.json

The file is a JSON array of model entries; see apps/control-api/examples/model_inventory.sample.json for a multi-backend example. If the file is missing or malformed, the API falls back to a built-in inventory so the control plane stays observable.

Ollama Backend Probe

Set OLLAMA_BASE_URL to point the control API at an Ollama backend:

export OLLAMA_BASE_URL=http://localhost:11434

The API exposes:

• GET /backends/ollama/health - backend reachability and status.
• GET /backends/ollama/models - model names returned by Ollama /api/tags.
• GET /backends/ollama/latency - lightweight latency measurement for /api/tags.

vLLM Backend Probe

Set VLLM_BASE_URL to point the control API at a vLLM OpenAI-compatible server:

export VLLM_BASE_URL=http://localhost:8000

The API exposes:

• GET /backends/vllm/health - backend reachability and status.
• GET /backends/vllm/models - model ids returned by vLLM /v1/models.
• GET /backends/vllm/latency - lightweight latency measurement for /v1/models.

Prometheus Metrics

GET /metrics exposes Prometheus-compatible metrics for request traffic, backend health, model inventory, capacity, and estimated cost.

Core metrics:

• ai_control_http_requests_total
• ai_control_http_request_latency_ms
• ai_control_backend_up
• ai_control_backend_latency_ms
• ai_control_model_available
• ai_control_capacity_available
• ai_control_estimated_hourly_cost_usd

AI Infrastructure Digital Twin

GET /topology exposes a live platform graph for private AI infrastructure components, dependencies, health, telemetry, and operational signals. The Ollama and vLLM nodes reflect live backend probe results (healthy/degraded plus measured latency). See docs/digital-twin.md.

GET /drift compares configured model inventory against live Ollama and vLLM probes and reports missing or unexpected models per backend.

POST /governance/evaluate runs a single AI request through workload identity, policy packs, prompt security, tenant quota, model registry, cost, risk, approval, sovereign AI, and live telemetry checks, then returns the final verdict for the operator dashboard playground and runtime enforcement path.

AI Cost Governance

governance/cost evaluates model usage, team budgets, token spend, and forecasted monthly cost into allow, warn, or block decisions.

AI Approval Workflow

governance/approval evaluates high-risk AI platform requests into allow, approval_required, or block decisions for human approval gates.

AI Governance Layer

governance/cost, governance/risk, and governance/approval model cost control, risk scoring, and human approval gates for private AI infrastructure.

governance/pipeline connects those signals into an end-to-end decision flow: request telemetry, cost decision, risk score, approval decision, and final verdict.

Run the demo pipeline:

python3.12 governance/pipeline/run_pipeline.py \
  --requests governance/pipeline/sample_requests.csv

Container Images

Every merge to main builds and pushes the control API image to GitHub Container Registry:

ghcr.io/justrunme/ai-infra-control-plane:latest
ghcr.io/justrunme/ai-infra-control-plane:

Images are signed with cosign and accompanied by an SPDX SBOM artifact from the release workflow. Tag releases with v* (for example v0.2.0) to publish semver tags.

Kubernetes Deployment

helm upgrade --install ai-control-plane infra/helm/ai-control-plane \
  --set image.repository=ghcr.io/justrunme/ai-infra-control-plane \
  --set image.tag=latest

The chart ships production defaults: non-root execution, read-only root filesystem, model inventory ConfigMap, HPA, PodDisruptionBudget, and optional ServiceMonitor, Ingress, and NetworkPolicy. See infra/helm/ai-control-plane/README.md.

Portfolio Docs

• docs/case-study.md explains the problem, architecture, capabilities, governance pipeline, observability, forecasting, GitOps, security, and demo flow.
• docs/platform-architecture.md describes the system boundary, logical layers, control API, governance architecture, delivery path, and extension points.

Remaining Backlog

See docs/product-roadmap.md for the current roadmap and maturity map. The next portfolio step is packaging evidence: record the enterprise demo GIF from docs/portfolio/demo-gif-script.md and add it to this README.

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: justrunme
• Source: justrunme/ai-infra-control-plane
• License: MIT

Install and usage instructions live in the source repository linked above.