Video Extract Agents

This project is a complete redesign of the original Video Extract tool.

Original implementation: https://github.com/cibis/video_extract

---

Video Extract Agents

A prompt-driven video extraction platform. Upload a video, describe what you want in plain English, and AI agents extract and compile the relevant segments into a highlight reel.

> Example: "Extract all kitesurfing jumps from this video and compile them into a highlight reel."

---

Why This Exists

This project exists because of two problems.

The first: I'm mildly obsessed with agentic AI, the idea that you give a system a goal in plain English and a crew of AI agents figures out how to get there. The second: I have hours of kitesurfing footage and zero patience for scrubbing through it frame by frame.

The obvious solution was to build an enterprise-grade, cloud-native, multi-agent video extraction platform.

So here we are: a full Azure microservices stack, CrewAI orchestration, MCP tool servers, and FFmpeg keyframe pipelines, all so I can type "find the jumps" and go back to the beach.

---

Table of Contents

• [Overview](#overview)
• [How It Works](#how-it-works)
• [Tech Stack](#tech-stack)
• [Quick Start (Local Dev)](#quick-start-local-dev)
• [Running Tests](#running-tests)
• [Documentation](#documentation)
• [External Agents](#external-agents)
• [Repository Structure](#repository-structure)

---

Overview

The platform combines agentic AI orchestration (CrewAI + Claude), MCP tool servers over SSE transport, and a cloud-native Azure microservices architecture to enable natural-language-driven video processing at scale.

Home — session active with completed job history and chat

Session History — completed and failed jobs with output files

Key capabilities:

• Upload videos up to 10 GB directly to Azure Blob Storage
• Describe what to extract in natural language via a chat interface
• AI agents (planner → analysis → processing) orchestrate the full pipeline
• FFmpeg keyframe pre-processing dramatically reduces AI token costs
• Output videos delivered via signed CDN URLs and email notification
• All processing services auto-scale to zero when idle (KEDA on Azure Container Apps)

---

How It Works

Upload video (Angular → Blob Storage via SAS token)
    ↓
Pre-processing worker (FFmpeg keyframe extraction → PostgreSQL index)
    ↓
User submits prompt (LibreChat iframe → API Gateway → Agent Orchestrator)
    ↓
CrewAI crew: Planner → Analysis Agent (MCP tools) → Processing Agent (MCP tools)
    ↓
Output video written to Blob Storage
    ↓
Signed download URL delivered via SSE stream + email notification

All steps are asynchronous and fault-tolerant via Azure Service Bus queues.

---

External Agents

The platform's MCP tools can be used directly from Claude Desktop or the LibreChat official image via an MCP bridge (port 8300) that translates standard MCP JSON-RPC to the platform's SSE tool protocol.

Claude Desktop:

# Start MCP bridge
bash external-agents/claude-desktop/scripts/start-mcp-bridge.sh

# Install config (Windows PowerShell)
.\external-agents\claude-desktop\scripts\install.ps1
# Restart Claude Desktop — Tools icon should show 19 tools

| Session started — upload link provided | Job complete — extraction summary and download link | |---|---| | | |

LibreChat (official image):

cp external-agents/librechat/.env.example external-agents/librechat/.env
# Set ANTHROPIC_API_KEY and generate random secrets (see docs/getting-started.md §13.2)
cd external-agents/librechat && docker compose up -d
# Open http://localhost:3081

| Agent running MCP tool calls (ingest → detect → clip) | Extraction complete — final output URL | |---|---| | | |

See [docs/getting-started.md § External agents](docs/getting-started.md#13-external-agents-librechat-official--claude-desktop) for the full walkthrough.

---

Tech Stack

| Layer | Technology | |---|---| | Frontend | Angular 19 + LibreChat (forked, iframe embed) | | API / BFF | Node.js + Express (TypeScript) | | AI Orchestration | Python + CrewAI + FastAPI | | LLM | Any LiteLLM-compatible model (Anthropic Claude, OpenAI, AWS Bedrock, and more) | | Tool Protocol | MCP over SSE transport | | Container Platform | Azure Container Apps + KEDA | | Infrastructure as Code | Terraform | | Storage | Azure Blob Storage | | Database | PostgreSQL 15 (ACA container, Azure Files backed) | | Messaging | Azure Service Bus | | Auth | Azure Entra External ID (magic link / JWT) | | Local Dev Emulation | Docker Compose + Azurite | | CI/CD | GitLab CI (mirrored to GitHub) |

---

Quick Start (Local Dev)

Requires Docker Desktop (≥ 4.30) with WSL 2. See [docs/getting-started.md](docs/getting-started.md) for full prerequisites and Azure setup.

1. Copy environment files:

cp backend/api-gateway/.env.example              backend/api-gateway/.env
cp backend/agent-orchestrator/.env.example       backend/agent-orchestrator/.env
cp backend/preprocessing-worker/.env.example     backend/preprocessing-worker/.env
cp mcp-servers/mcp-server-analysis/.env.example  mcp-servers/mcp-server-analysis/.env
cp mcp-servers/mcp-server-processing/.env.example mcp-servers/mcp-server-processing/.env
cp frontend/librechat/.env.example               frontend/librechat/.env

Edit backend/agent-orchestrator/.env and set ANTHROPIC_API_KEY.

2. Start the stack:

cd infrastructure/docker-compose
docker compose up --build

3. Create Service Bus queues (once, after stack is up):

export SERVICE_BUS_CONNECTION_STRING="Endpoint=sb://localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;"
python scripts/create_service_bus_queues.py

4. Verify services:

curl http://localhost:8000/health   # API Gateway
curl http://localhost:8001/health   # Agent Orchestrator
curl http://localhost:8100/tools    # MCP Analysis tools
curl http://localhost:8200/tools    # MCP Processing tools

Services run on:

| Service | Port | |---|---| | Angular Shell | http://localhost:4200 | | LibreChat | http://localhost:3080 | | API Gateway | http://localhost:8000 | | Agent Orchestrator | http://localhost:8001 | | MCP Analysis | http://localhost:8100 | | MCP Processing | http://localhost:8200 | | Azurite (Blob) | http://localhost:10000 | | PostgreSQL | localhost:5433 |

---

Running Tests

E2E tests (fully containerised)

scripts/run-e2e-local.sh
# With frontier vision tools:
ANTHROPIC_API_KEY=sk-... scripts/run-e2e-local.sh

---

Documentation

| Document | Description | |---|---| | [docs/architecture.md](docs/architecture.md) | System design, data flows, service responsibilities, component details, deployment diagrams | | [docs/getting-started.md](docs/getting-started.md) | Full setup guide — prerequisites, GitLab/GitHub/Azure configuration, local dev bootstrap, CI/CD variables, secrets reference, troubleshooting | | [docs/local-development.md](docs/local-development.md) | Day-to-day local development — starting the stack, running services and tests, common tasks | | [docs/e2e-tests.md](docs/e2e-tests.md) | End-to-end pipeline tests | | [docs/azure-production-deployment.md](docs/azure-production-deployment.md) | Azure production deployment reference — services, roles, inter-service communication, scaling, CI/CD | | [docs/azure-credentials.md](docs/azure-credentials.md) | Azure credentials setup — every credential the platform needs, how to create and configure each | | [docs/terraform.md](docs/terraform.md) | Terraform layout, modules, environments, and how the pieces connect | | [docs/ai-containers-deep-dive.md](docs/ai-containers-deep-dive.md) | Deep dive into each AI container — inputs, outputs, and position in the job processing sequence | | [docs/gitlab-pipeline.md](docs/gitlab-pipeline.md) | CI/CD pipeline — every stage and job, environment lifecycle, and SDLC workflow | | [docs/instant-compilation-errors.md](docs/instant-compilation-errors.md) | Getting immediate type and syntax error feedback during local development without Docker rebuilds | | [docs/local-containers-report.md](docs/local-containers-report.md) | Local container architecture report | | [external-agents/claude-desktop/README.md](external-agents/claude-desktop/README.md) | Claude Desktop MCP integration | | [external-agents/librechat/README.md](external-agents/librechat/README.md) | LibreChat official image MCP integration |

---

Repository Structure

backend/
  api-gateway/            Node.js + Express (TypeScript) — auth, SAS tokens, SSE, chat proxy
  agent-orchestrator/     Python + CrewAI (FastAPI) — planner/analyst/processor agents
  preprocessing-worker/   Python — FFmpeg keyframe extraction

mcp-servers/
  mcp-server-analysis/    Port 8100 — ingest_video, extract_frames, detect_motion, detect_motion_sports,
                          detect_objects, detect_objects_vision, analyze_scene, transcribe_audio,
                          estimate_height_above_surface, read_asset, query_asset, write_query_asset,
                          write_segments_asset
  mcp-server-processing/  Port 8200 — split_video, extract_clip, extract_clips_bulk, merge_clips,
                          transform_video, write_asset, query_asset, write_query_asset

frontend/
  angular-shell/          Angular 19 — upload UI, job dashboard, LibreChat iframe host
  librechat/              Forked LibreChat — custom endpoint, branding, job status postMessage bridge

external-agents/
  mcp-bridge/             Standard MCP server (port 8300) — SSE + stdio transports
  claude-desktop/         Claude Desktop config + install scripts
  librechat/              LibreChat official image stack
  agent-instructions/     System prompt for external agents

infrastructure/
  docker-compose/         Full local dev stack
  terraform/
    modules/              aca, storage, database (reusable modules)
    envs/                 dev, test (ephemeral per CI pipeline)

tests/
  e2e/                    End-to-end tests (ephemeral Azure + local Docker Compose)

scripts/
  init_db.py                    Create all database tables
  init_storage.py               Create Blob Storage containers
  create_service_bus_queues.py  Create all Service Bus queues
  run-e2e-local.sh              Run E2E tests locally (fully containerised)
  bootstrap-dev.sh              First-time local dev setup
  smoke-test.sh                 Quick smoke test against running stack
  teardown.sh                   Stop and clean up local stack
  repair_job_output.py          Repair job output records in PostgreSQL
  collect_test_logs.py          Collect logs from CI test run
  check_e2e_threshold.py        Assert E2E test pass rate meets threshold

docs/
  architecture.md         System architecture reference
  getting-started.md      Full setup and deployment guide

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: cibis
• Source: cibis/video-extract-agents
• License: MIT

Install and usage instructions live in the source repository linked above.