# Livekit Agent Simulator

> Black-box LiveKit voice agent testing: real WebRTC/SIP calls with an AI caller. Catch barge-in, noise, quiet-mic, and latency failures that text pytest misses. Forensic reports + MCP + lks CLI — no agent code changes.

- **Type:** MCP server
- **Install:** `agentstack add mcp-quangdang46-livekit-agent-simulator`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [quangdang46](https://agentstack.voostack.com/s/quangdang46)
- **Installs:** 0
- **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [quangdang46](https://github.com/quangdang46)
- **Source:** https://github.com/quangdang46/livekit_agent_simulator

## Install

```sh
agentstack add mcp-quangdang46-livekit-agent-simulator
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# livekit-agent-simulator

  

[](https://opensource.org/licenses/MIT)
[](https://github.com/quangdang46/livekit_agent_simulator/actions/workflows/ci.yml)
[](https://github.com/quangdang46/livekit_agent_simulator/releases)

**Dial any LiveKit voice agent with an AI simulated caller — WebRTC room, inbound SIP, or outbound SIP — and keep a full forensic log.**  
Standalone MCP server + CLI (`lks`). Black-box testing: no imports from the agent under test, no edits to its code or `.env`.

Quick Install

```bash
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify
```

### Install via coding agent (copy-paste)

Paste into Claude Code, Cursor, Codex, AmpCode, Windsurf, or any coding agent **from the repo you want to test**:

```text
Install and configure livekit-agent-simulator (CLI: lks) for this project by following the instructions here:
https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/docs/guide/installation.md

Target project root is this workspace. Use absolute --root paths. Install the portable CLI if missing, run lks init, help fill .agent-sim/config.yaml from my local env or ask me for LiveKit + active caller provider key (Gemini Live or OpenAI Realtime) + agent_name, ensure .agent-sim is gitignored, run preflight, and stop before execute if the voice agent worker is not running. Do not edit agent application source outside .agent-sim/.
```

Same idea, one line:

```text
Install and configure livekit-agent-simulator by following: https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/docs/guide/installation.md
```

---

## TL;DR

### The Problem

Voice agents fail in ways unit tests never see:

| Gap | What you miss |
|-----|----------------|
| No real caller | Scripts that never interrupt, stall, or switch language |
| Chat-only evals | No room events, audio timing, or tool spans |
| Manual QA calls | Not CI-reproducible, no structured PassCriteria |
| Agent-coupled harnesses | Tests break when you refactor the worker |

### The Solution

**livekit-agent-simulator** drives an AI simulated caller — Gemini Live or OpenAI Realtime (per `simulator.provider`) — from scenario JSONL over one of three transport modes (`Caller.mode`), observes transcripts / tools / flow / room events, and writes a timestamped report you can play back.

| Surface | What you get |
|---------|--------------|
| `lks` CLI | init → preflight → execute → report → web |
| MCP server | Same ops for Claude Code, Cursor, Codex, … |
| Transport modes | `webrtc_sim` · `inbound_sip` · `outbound_human_pickup` · `outbound_sim_callee` (optional `agent_dials`) |
| Reports | `events.jsonl`, `timeline.md`, `summary.json`, optional stereo WAV |
| Judge | Optional LLM PassCriteria scoring |

### Why Use lks?

| Feature | What it does |
|---------|--------------|
| **Black-box dispatch** | Only needs `agent_name` + LiveKit creds |
| **5 transport modes** | WebRTC · inbound SIP · outbound human pickup · outbound sim callee · agent_dials |
| **Scenario JSONL** | Persona, Caller, Telephony, Execute, Script, PassCriteria, Dispatch |
| **Forensic log** | Per-turn events in SQLite + `reports//` |
| **Report player** | Local web UI: audio + transcript sync |
| **CLI ↔ MCP parity** | One `ops` layer — no duplicate run paths |
| **Portable packs** | Download installer; no uv/pip required for users |

---

### Quick Example

```bash
# Install once
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

# In the repo you want to test (agent worker must already be running)
lks init --root /path/to/target
# edit /path/to/target/.agent-sim/config.yaml  (LiveKit + active provider keys, agent_name)

lks preflight --root /path/to/target
lks execute smoke-hello --root /path/to/target
lks report  --root /path/to/target
lks web --root /path/to/target          # Ctrl+C to stop
```

---

## Design Philosophy

1. **The agent under test is a black box.**  
   We never import or patch target application code. Dispatch metadata is opaque JSON.

2. **Generic core, target-owned config.**  
   Language, timezone, topics, and business strings belong in the target’s `.agent-sim/` — not hardcoded in the package.

3. **One ops layer for CLI and MCP.**  
   `execute_*` validates then runs. No “run vs execute” forks.

4. **Forensics over vibes.**  
   Every run produces structured events you can `compare`, `log`, and play back.

5. **CI-friendly gates.**  
   Hard fails on status / assert / script; optional strict judge for softer LLM scoring.

---

## How It Works

```text
1. Read /.agent-sim/config.yaml
2. Pick SimLeg from scenario Caller.mode (webrtc_sim | inbound_sip | outbound_human_pickup | outbound_sim_callee | agent_dials)
3. Connect leg → LiveKit room(s) / SIP hairpin as needed; the active caller provider (Gemini Live / OpenAI Realtime) stays WebRTC in the sim room
4. Bridge audio; observe transcripts, tools, timing, interruptions
5. Write reports// + runs.sqlite
6. Optional LLM judge vs PassCriteria
```

```text
                    Caller.mode (scenario)
         ┌───────────────┬────────────────┬──────────────────┬────────────────────┐
         │  webrtc_sim   │  inbound_sip   │   outbound_human_pickup   │ outbound_sim_callee│
         │  room audio   │  sim dials DID │ human answers → │  Gemini SIP callee │
         │               │                │ Gemini colocated│  (2-room hairpin)  │
         └───────┬───────┴────────┬───────┴────────┬─────────┴─────────┬──────────┘
                 │                │                │                   │
                 └────────────────┼────────────────┼───────────────────┘
                                  ▼
                    ┌──────────────────────────┐
                    │  Sim caller persona      │
                    │  (Gemini Live / OpenAI   │
                    │   Realtime) + LiveKit    │
                    │   agent (black box)      │
                    └────────────┬─────────────┘
                                 │ observe
                                 ▼
                    reports// · runs.sqlite · judge
```

Mode details and config: [docs/telephony.md](docs/telephony.md). Templates: `inbound-caller-sim`, `outbound-human-pickup`, `outbound-callee-sim`.

---

## How lks Compares

| Approach | Real room | AI caller | Forensic log | MCP | Black-box |
|----------|-----------|-----------|--------------|-----|-----------|
| Manual phone QA | ✅ | ❌ | ❌ | ❌ | ✅ |
| Unit / mock STT | ❌ | ❌ | Partial | ❌ | ❌ |
| In-repo agent tests | ⚠️ | ⚠️ | Varies | ❌ | Often coupled |
| **lks** | ✅ LiveKit | ✅ Gemini Live / OpenAI Realtime | ✅ Full | ✅ | ✅ |

**When to use lks:**
- Regression suites for LiveKit voice agents
- Agent-driven CI / coding-agent workflows (MCP)
- Debugging turn-taking, tools, and silence without reading agent source

**When it might not be ideal:**
- Pure text chatbots with no LiveKit room
- Offline environments without LiveKit + an active caller provider API (Gemini Live or OpenAI Realtime)

---

## Installation

### Quick install (recommended)

**Download only — no uv/pip/build on your machine.** CI ships a portable pack (embedded Python + deps + report player).

```bash
# macOS / Linux
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify
```

```powershell
# Windows PowerShell
irm "https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/install.ps1" -OutFile "$env:TEMP\lks-install.ps1"
powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\lks-install.ps1" -Verify
```

Also available from a release asset:

```bash
curl -fsSL "https://github.com/quangdang46/livekit_agent_simulator/releases/download/v0.1.0/install.sh" \
  | bash -s -- --verify
```

| Flag | Purpose |
|------|---------|
| `--verify` | Checksum verification |
| `--ref v0.1.0` | Pin release tag |
| `--no-mcp` | Skip MCP registration into coding tools |
| `--uninstall` | Remove install |

By default the installer registers the MCP server `livekit-agent-simulator` (`lks mcp`) into detected tools: Claude Code, Cursor, Cline, Windsurf, VS Code Copilot, Gemini CLI, Amazon Q, OpenCode, Codex, Warp.

**Agent-oriented install playbook (long form):** [docs/guide/installation.md](docs/guide/installation.md)  
Raw URL for paste into agents:  
`https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/docs/guide/installation.md`

### From source (maintainers / contributors)

```bash
git clone https://github.com/quangdang46/livekit_agent_simulator.git
cd livekit-agent-simulator
uv sync --extra dev
uv run lks --help
```

Requires **Python 3.10–3.13**.

### Web UI (maintainers)

Users never build this — CI packs `web/dist` into the wheel as `web_static`. Source: `web/`.

```bash
pnpm --dir web install
pnpm --dir web build                    # → web/dist/ (attached by Hatch on uv build)
pnpm --dir web dev                      # HMR; proxy /api + /runs → lks web :8765
```

See `web/README.md`.

---

## Quick Start

```bash
# Agent worker must be running and registered with LiveKit
lks guide
lks init --root /path/to/target
# fill .agent-sim/config.yaml

lks preflight --root /path/to/target
lks scenario-init smoke-hello --root /path/to/target   # if needed
lks validate smoke-hello --root /path/to/target
lks execute smoke-hello --root /path/to/target
lks runs --root /path/to/target
lks report  --root /path/to/target
lks web --root /path/to/target
```

### Minimal scenario (`smoke-hello`)

```yaml
apiVersion: agent-sim/v1
kind: Scenario
metadata:
  id: smoke-hello
  locale: en-US
  tags: [smoke]
persona:
  name: Alex
  brief: First-time caller; confirm you reached the right place, then end politely.
  goals:
  - Hear the agent
  - Say you will call back
  style: polite, brief
execute:
  max_turns: 2
  timeout_s: 90
  first_speaker: user
pass_criteria:
  criteria:
  - The agent responded to the caller
  - The agent responded in the caller's language
```

Optional multi-judge PassCriteria: `judges[]` + `mode` (`all` \| `majority` \| `any`). Assert highlights (`tool_order`, `constraint_respected`, recovery/latency): `lks guide`.

Full-line `#` comments in scaffolded YAML are guides — runtime ignores them. Legacy `*.jsonl` scenarios are still read.

---

## Configuration

Target-only data lives under `/.agent-sim/` (**gitignored**). Created by `init`.

| Section | Required | Purpose |
|---------|----------|---------|
| `livekit.url` | yes | `wss://…` LiveKit Cloud or self-host |
| `livekit.api_key` / `api_secret` | yes | Server API credentials |
| `livekit.agent_name` | yes | Must match worker dispatch name |
| `livekit.dispatch_metadata` | no | Default opaque JSON **string** for all runs |
| `simulator.api_key` | yes | Key of the **active** caller provider (`google` → Gemini, `openai` → OpenAI) |
| `simulator.provider` / `mode` | no | Caller brain: `google` (default) or `openai`; `realtime` mode (cascade reserved) |
| `simulator.voice.model` / `voice` / `language` | no | Provider-neutral voice bag; defaults flash-live, Puck, `en-US` |
| `simulator.profiles` | no | **Named caller profiles** — switch provider without editing the file |
| `judge.model` | no | If set + PassCriteria → post-run LLM judge |
| `observe.record_audio` | no (default `true`) | Local stereo WAV (L=sim, R=agent); no Egress |
| `observe.data_topics` | no | Empty = all topics |
| `observe.tool_event_patterns` | no | Map data payloads → tool start/end/error |

See template: [`templates/config.yaml`](templates/config.yaml). Consumer-specific wiring: [`docs/portability.md`](docs/portability.md).

### Switching caller provider with `--profile`

To A/B test the same scenario against **Gemini Live** vs **OpenAI Realtime** (or
any set of provider/voice combos) **without editing `config.yaml` between runs**,
define named profiles under `simulator.profiles:` and select one with
`--profile ` on `execute` / `execute-all` / `preflight`.

```yaml
simulator:
  # legacy flat block = fallback (used when no --profile flag and no default profile)
  provider: google
  mode: realtime
  api_key: "AQ.Ab8..."                 # Gemini Live key

  # named profiles — switch with --profile 
  profiles:
    gemini:
      default: true                    # auto-selected when no --profile flag
      provider: google
      api_key: "AQ.Ab8..."             # Gemini Live key
    openai:
      provider: openai
      api_key: "sk-..."                # OpenAI key
      voice:
        model: "gpt-realtime-2.1-mini"
        voice: "marin"
```

```bash
lks execute smoke-hello                    # `gemini` (marked default: true)
lks execute smoke-hello --profile gemini   # Gemini Live caller
lks execute smoke-hello --profile openai   # OpenAI Realtime caller
```

**Selection** (`--profile` absent): if **exactly one** profile has
`default: true`, it is used; otherwise the legacy flat `simulator:` block runs.
2+ profiles marked `default: true` is an error (no "first wins"). If `profiles:`
exist with **no** default and **no** flat-block credentials, config loading
errors loudly (no silent fallback). `--profile ` always wins regardless
of which profile is default. A missing profile name fails loudly (lists
available profiles) — no silent fallback. Profile names are **case-sensitive**.

**Precedence:** profile field → flat `simulator:` field → built-in default. A
profile **inherits** unspecified fields (voice, language, mode) from the flat
block, so `openai` above only overrides `provider` + `api_key` + `voice`, and
keeps `mode: realtime`. Presence of `profiles:` never changes what runs when
neither `--profile` nor a `default: true` profile is present — that is the flat
block (backward compatible).

> **⚠️ Gemini caller model note (observed 2026-08):**
> `gemini-3.1-flash-live-preview` — the historical default — is a **preview** model with known instability as the simulated caller: transient mid-call WebSocket drops (`APIError 1006 / 1008`, end reason `gemini_socket_drop`) in ~2/15 real runs, plus LiveKit-documented limits (`send_client_content` rejected after the first model turn, `update_instructions`/`generate_reply` unsupported). If you see calls ending with `gemini_socket_drop`, switch the caller model to a stable release, e.g.:
> ```yaml
> simulator:
>   voice:
>     model: "gemini-2.5-flash-native-audio-preview-12-2025"   # or gemini-live-2.5-flash-native-audio (GA)
> ```
> Verified: `gemini-2.5-flash-native-audio-preview-12-2025` connects and talks as the caller with **0 socket drops** across real runs (the `-12-2025` date suffix is required — `gemini-2.5-flash-native-audio-preview` alone returns `API_KEY_INVALID`).

---

## Commands

CLI and MCP share the same public ops (`ops.py`). Prefer `execute` (validate then run).

| CLI | MCP tool | Purpose |
|-----|----------|---------|
| `init` | `init_project` | Scaffold `.agent-sim/` + gitignore |
| `guide` | `guide` | Setup/ops guide (markdown) |
| `web` | `web` | Local report player |
| `preflight` | `preflight` | Config + LiveKit connectivity |
| `scenarios` | `list_scenarios` | List `scenarios/*.yaml` (legacy `*.jsonl` read) |
| `plugins` | `list_plugins` | Verify plugins |
| `cues` | `list_cues` | Built-in + local PCM cues |
| `validate` | `validate_scenario` | Schema + lint |
| `export` | `export_scenario` | Parsed scenario JSON |
| `scenario-init` | `init_scenario` | Scaffold JSONL with `//` guides |
| `execute` | `execute_scenario` | Validate then run one scenario |
| `execute-all` | `execute_scenarios` | Batch (ids / tag) |
| `execute-dict` | `execute_scenario_dict` | In-memory scenario dict |
| `status` | `get_run_status` | SQLite run status |
| `log` | `get_run_log` | Filtered `events.jsonl` |
| `report` | `get_run_report` | Summary + verdict + paths |
| `compare` | `compare_runs` | Diff two runs; `--baseline` hard-fails on latenc

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [quangdang46](https://github.com/quangdang46)
- **Source:** [quangdang46/livekit_agent_simulator](https://github.com/quangdang46/livekit_agent_simulator)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-quangdang46-livekit-agent-simulator
- Seller: https://agentstack.voostack.com/s/quangdang46
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.