Livekit Agent Simulator

livekit-agent-simulator

[](https://opensource.org/licenses/MIT) [](https://github.com/quangdang46/livekitagentsimulator/actions/workflows/ci.yml) [](https://github.com/quangdang46/livekitagentsimulator/releases)

Dial any LiveKit voice agent with an AI simulated caller — WebRTC room, inbound SIP, or outbound SIP — and keep a full forensic log. Standalone MCP server + CLI (lks). Black-box testing: no imports from the agent under test, no edits to its code or .env.

Quick Install

curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

Install via coding agent (copy-paste)

Paste into Claude Code, Cursor, Codex, AmpCode, Windsurf, or any coding agent from the repo you want to test:

Install and configure livekit-agent-simulator (CLI: lks) for this project by following the instructions here:
https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/docs/guide/installation.md

Target project root is this workspace. Use absolute --root paths. Install the portable CLI if missing, run lks init, help fill .agent-sim/config.yaml from my local env or ask me for LiveKit + active caller provider key (Gemini Live or OpenAI Realtime) + agent_name, ensure .agent-sim is gitignored, run preflight, and stop before execute if the voice agent worker is not running. Do not edit agent application source outside .agent-sim/.

Same idea, one line:

Install and configure livekit-agent-simulator by following: https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/docs/guide/installation.md

---

TL;DR

The Problem

Voice agents fail in ways unit tests never see:

| Gap | What you miss | |-----|----------------| | No real caller | Scripts that never interrupt, stall, or switch language | | Chat-only evals | No room events, audio timing, or tool spans | | Manual QA calls | Not CI-reproducible, no structured PassCriteria | | Agent-coupled harnesses | Tests break when you refactor the worker |

The Solution

livekit-agent-simulator drives an AI simulated caller — Gemini Live or OpenAI Realtime (per simulator.provider) — from scenario JSONL over one of three transport modes (Caller.mode), observes transcripts / tools / flow / room events, and writes a timestamped report you can play back.

| Surface | What you get | |---------|--------------| | lks CLI | init → preflight → execute → report → web | | MCP server | Same ops for Claude Code, Cursor, Codex, … | | Transport modes | webrtc_sim · inbound_sip · outbound_human_pickup · outbound_sim_callee (optional agent_dials) | | Reports | events.jsonl, timeline.md, summary.json, optional stereo WAV | | Judge | Optional LLM PassCriteria scoring |

Why Use lks?

| Feature | What it does | |---------|--------------| | Black-box dispatch | Only needs agent_name + LiveKit creds | | 5 transport modes | WebRTC · inbound SIP · outbound human pickup · outbound sim callee · agent_dials | | Scenario JSONL | Persona, Caller, Telephony, Execute, Script, PassCriteria, Dispatch | | Forensic log | Per-turn events in SQLite + reports// | | Report player | Local web UI: audio + transcript sync | | CLI ↔ MCP parity | One ops layer — no duplicate run paths | | Portable packs | Download installer; no uv/pip required for users |

---

Quick Example

# Install once
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

# In the repo you want to test (agent worker must already be running)
lks init --root /path/to/target
# edit /path/to/target/.agent-sim/config.yaml  (LiveKit + active provider keys, agent_name)

lks preflight --root /path/to/target
lks execute smoke-hello --root /path/to/target
lks report  --root /path/to/target
lks web --root /path/to/target          # Ctrl+C to stop

---

Design Philosophy

1. The agent under test is a black box.

We never import or patch target application code. Dispatch metadata is opaque JSON.

1. Generic core, target-owned config.

Language, timezone, topics, and business strings belong in the target’s .agent-sim/ — not hardcoded in the package.

1. One ops layer for CLI and MCP.

execute_* validates then runs. No “run vs execute” forks.

1. Forensics over vibes.

Every run produces structured events you can compare, log, and play back.

1. CI-friendly gates.

Hard fails on status / assert / script; optional strict judge for softer LLM scoring.

---

How It Works

1. Read /.agent-sim/config.yaml
2. Pick SimLeg from scenario Caller.mode (webrtc_sim | inbound_sip | outbound_human_pickup | outbound_sim_callee | agent_dials)
3. Connect leg → LiveKit room(s) / SIP hairpin as needed; the active caller provider (Gemini Live / OpenAI Realtime) stays WebRTC in the sim room
4. Bridge audio; observe transcripts, tools, timing, interruptions
5. Write reports// + runs.sqlite
6. Optional LLM judge vs PassCriteria

                    Caller.mode (scenario)
         ┌───────────────┬────────────────┬──────────────────┬────────────────────┐
         │  webrtc_sim   │  inbound_sip   │   outbound_human_pickup   │ outbound_sim_callee│
         │  room audio   │  sim dials DID │ human answers → │  Gemini SIP callee │
         │               │                │ Gemini colocated│  (2-room hairpin)  │
         └───────┬───────┴────────┬───────┴────────┬─────────┴─────────┬──────────┘
                 │                │                │                   │
                 └────────────────┼────────────────┼───────────────────┘
                                  ▼
                    ┌──────────────────────────┐
                    │  Sim caller persona      │
                    │  (Gemini Live / OpenAI   │
                    │   Realtime) + LiveKit    │
                    │   agent (black box)      │
                    └────────────┬─────────────┘
                                 │ observe
                                 ▼
                    reports// · runs.sqlite · judge

Mode details and config: [docs/telephony.md](docs/telephony.md). Templates: inbound-caller-sim, outbound-human-pickup, outbound-callee-sim.

---

How lks Compares

| Approach | Real room | AI caller | Forensic log | MCP | Black-box | |----------|-----------|-----------|--------------|-----|-----------| | Manual phone QA | ✅ | ❌ | ❌ | ❌ | ✅ | | Unit / mock STT | ❌ | ❌ | Partial | ❌ | ❌ | | In-repo agent tests | ⚠️ | ⚠️ | Varies | ❌ | Often coupled | | lks | ✅ LiveKit | ✅ Gemini Live / OpenAI Realtime | ✅ Full | ✅ | ✅ |

When to use lks:

• Regression suites for LiveKit voice agents
• Agent-driven CI / coding-agent workflows (MCP)
• Debugging turn-taking, tools, and silence without reading agent source

When it might not be ideal:

• Pure text chatbots with no LiveKit room
• Offline environments without LiveKit + an active caller provider API (Gemini Live or OpenAI Realtime)

---

Installation

Quick install (recommended)

Download only — no uv/pip/build on your machine. CI ships a portable pack (embedded Python + deps + report player).

# macOS / Linux
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/install.sh?$(date +%s)" \
  | bash -s -- --verify

# Windows PowerShell
irm "https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/install.ps1" -OutFile "$env:TEMP\lks-install.ps1"
powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\lks-install.ps1" -Verify

Also available from a release asset:

curl -fsSL "https://github.com/quangdang46/livekit_agent_simulator/releases/download/v0.1.0/install.sh" \
  | bash -s -- --verify

| Flag | Purpose | |------|---------| | --verify | Checksum verification | | --ref v0.1.0 | Pin release tag | | --no-mcp | Skip MCP registration into coding tools | | --uninstall | Remove install |

By default the installer registers the MCP server livekit-agent-simulator (lks mcp) into detected tools: Claude Code, Cursor, Cline, Windsurf, VS Code Copilot, Gemini CLI, Amazon Q, OpenCode, Codex, Warp.

Agent-oriented install playbook (long form): [docs/guide/installation.md](docs/guide/installation.md) Raw URL for paste into agents: https://raw.githubusercontent.com/quangdang46/livekit_agent_simulator/main/docs/guide/installation.md

From source (maintainers / contributors)

git clone https://github.com/quangdang46/livekit_agent_simulator.git
cd livekit-agent-simulator
uv sync --extra dev
uv run lks --help

Requires Python 3.10–3.13.

Web UI (maintainers)

Users never build this — CI packs web/dist into the wheel as web_static. Source: web/.

pnpm --dir web install
pnpm --dir web build                    # → web/dist/ (attached by Hatch on uv build)
pnpm --dir web dev                      # HMR; proxy /api + /runs → lks web :8765

See web/README.md.

---

Quick Start

# Agent worker must be running and registered with LiveKit
lks guide
lks init --root /path/to/target
# fill .agent-sim/config.yaml

lks preflight --root /path/to/target
lks scenario-init smoke-hello --root /path/to/target   # if needed
lks validate smoke-hello --root /path/to/target
lks execute smoke-hello --root /path/to/target
lks runs --root /path/to/target
lks report  --root /path/to/target
lks web --root /path/to/target

Minimal scenario (smoke-hello)

apiVersion: agent-sim/v1
kind: Scenario
metadata:
  id: smoke-hello
  locale: en-US
  tags: [smoke]
persona:
  name: Alex
  brief: First-time caller; confirm you reached the right place, then end politely.
  goals:
  - Hear the agent
  - Say you will call back
  style: polite, brief
execute:
  max_turns: 2
  timeout_s: 90
  first_speaker: user
pass_criteria:
  criteria:
  - The agent responded to the caller
  - The agent responded in the caller's language

Optional multi-judge PassCriteria: judges[] + mode (all \| majority \| any). Assert highlights (tool_order, constraint_respected, recovery/latency): lks guide.

Full-line # comments in scaffolded YAML are guides — runtime ignores them. Legacy *.jsonl scenarios are still read.

---

Configuration

Target-only data lives under /.agent-sim/ (gitignored). Created by init.

| Section | Required | Purpose | |---------|----------|---------| | livekit.url | yes | wss://… LiveKit Cloud or self-host | | livekit.api_key / api_secret | yes | Server API credentials | | livekit.agent_name | yes | Must match worker dispatch name | | livekit.dispatch_metadata | no | Default opaque JSON string for all runs | | simulator.api_key | yes | Key of the active caller provider (google → Gemini, openai → OpenAI) | | simulator.provider / mode | no | Caller brain: google (default) or openai; realtime mode (cascade reserved) | | simulator.voice.model / voice / language | no | Provider-neutral voice bag; defaults flash-live, Puck, en-US | | simulator.profiles | no | Named caller profiles — switch provider without editing the file | | judge.model | no | If set + PassCriteria → post-run LLM judge | | observe.record_audio | no (default true) | Local stereo WAV (L=sim, R=agent); no Egress | | observe.data_topics | no | Empty = all topics | | observe.tool_event_patterns | no | Map data payloads → tool start/end/error |

See template: [templates/config.yaml](templates/config.yaml). Consumer-specific wiring: [docs/portability.md](docs/portability.md).

Switching caller provider with --profile

To A/B test the same scenario against Gemini Live vs OpenAI Realtime (or any set of provider/voice combos) without editing config.yaml between runs, define named profiles under simulator.profiles: and select one with --profile on execute / execute-all / preflight.

simulator:
  # legacy flat block = fallback (used when no --profile flag and no default profile)
  provider: google
  mode: realtime
  api_key: "AQ.Ab8..."                 # Gemini Live key

  # named profiles — switch with --profile 
  profiles:
    gemini:
      default: true                    # auto-selected when no --profile flag
      provider: google
      api_key: "AQ.Ab8..."             # Gemini Live key
    openai:
      provider: openai
      api_key: "sk-..."                # OpenAI key
      voice:
        model: "gpt-realtime-2.1-mini"
        voice: "marin"

lks execute smoke-hello                    # `gemini` (marked default: true)
lks execute smoke-hello --profile gemini   # Gemini Live caller
lks execute smoke-hello --profile openai   # OpenAI Realtime caller

Selection (--profile absent): if exactly one profile has default: true, it is used; otherwise the legacy flat simulator: block runs. 2+ profiles marked default: true is an error (no "first wins"). If profiles: exist with no default and no flat-block credentials, config loading errors loudly (no silent fallback). --profile always wins regardless of which profile is default. A missing profile name fails loudly (lists available profiles) — no silent fallback. Profile names are case-sensitive.

Precedence: profile field → flat simulator: field → built-in default. A profile inherits unspecified fields (voice, language, mode) from the flat block, so openai above only overrides provider + api_key + voice, and keeps mode: realtime. Presence of profiles: never changes what runs when neither --profile nor a default: true profile is present — that is the flat block (backward compatible).

> ⚠️ Gemini caller model note (observed 2026-08): > gemini-3.1-flash-live-preview — the historical default — is a preview model with known instability as the simulated caller: transient mid-call WebSocket drops (APIError 1006 / 1008, end reason gemini_socket_drop) in ~2/15 real runs, plus LiveKit-documented limits (send_client_content rejected after the first model turn, update_instructions/generate_reply unsupported). If you see calls ending with gemini_socket_drop, switch the caller model to a stable release, e.g.: > ``yaml > simulator: > voice: > model: "gemini-2.5-flash-native-audio-preview-12-2025" # or gemini-live-2.5-flash-native-audio (GA) > ` > Verified: gemini-2.5-flash-native-audio-preview-12-2025 connects and talks as the caller with **0 socket drops** across real runs (the -12-2025 date suffix is required — gemini-2.5-flash-native-audio-preview alone returns APIKEYINVALID`).

---

Commands

CLI and MCP share the same public ops (ops.py). Prefer execute (validate then run).

| CLI | MCP tool | Purpose | |-----|----------|---------| | init | init_project | Scaffold .agent-sim/ + gitignore | | guide | guide | Setup/ops guide (markdown) | | web | web | Local report player | | preflight | preflight | Config + LiveKit connectivity | | scenarios | list_scenarios | List scenarios/*.yaml (legacy *.jsonl read) | | plugins | list_plugins | Verify plugins | | cues | list_cues | Built-in + local PCM cues | | validate | validate_scenario | Schema + lint | | export | export_scenario | Parsed scenario JSON | | scenario-init | init_scenario | Scaffold JSONL with // guides | | execute | execute_scenario | Validate then run one scenario | | execute-all | execute_scenarios | Batch (ids / tag) | | execute-dict | execute_scenario_dict | In-memory scenario dict | | status | get_run_status | SQLite run status | | log | get_run_log | Filtered events.jsonl | | report | get_run_report | Summary + verdict + paths | | compare | compare_runs | Diff two runs; --baseline hard-fails on latenc

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: quangdang46
• Source: quangdang46/livekitagent_simulator
• License: MIT

Install and usage instructions live in the source repository linked above.