Build Voice Agent

Build a voice agent with Patter

Patter is an open-source SDK that turns any AI agent into a phone agent. Pick one of three architectures (Realtime, ConvAI, Pipeline), describe the behaviour in a system prompt, and run a local server that connects to your Twilio or Telnyx number — no Patter Cloud, no managed service. The four-line pattern below is the entire surface.

Architecture in one diagram

┌────────────────────────────────────────────────────────────────────┐
│                          Patter server (local)                      │
│   ┌────────────────┐    ┌───────────────────┐    ┌───────────────┐ │
│   │ Carrier WS     │───▶│  Agent (engine /  │───▶│  Carrier WS   │ │
│   │ in (audio in)  │    │  pipeline) loop   │    │ out (TTS out) │ │
│   └────────────────┘    └───────────────────┘    └───────────────┘ │
│           ▲                                              │          │
│           │                Built-in tools:               │          │
│           │             transfer_call · end_call         │          │
└───────────┼──────────────────────────────────────────────┼──────────┘
            │                                              │
            │ WSS (mulaw 8kHz / pcm 16kHz)                 │
            ▼                                              ▼
       ┌──────────┐                                   ┌──────────┐
       │  Twilio  │  ────────  real phone call  ───── │   User   │
       │  Telnyx  │                                   └──────────┘
       └──────────┘

You write: system prompt, optional tools, optional guardrails, choice of engine. Patter writes: WebSocket framing, audio transcoding, barge-in, VAD, cost tracking, tunnel, dashboard.

Decide the mode

| Mode | When to use | Latency | Cost | |---|---|---|---| | OpenAIRealtime2 (GA, default) | Default for everything. Lowest latency, single key. | ~200 ms turn | $$$ | | OpenAIRealtime | Legacy gpt-realtime-mini; pick only if explicitly required. | ~200 ms turn | $$ | | ElevenLabsConvAI | Turn-taking conversations (slower but better at long pauses). | ~600 ms turn | $$ | | Pipeline | Custom STT/LLM/TTS combinations. Use when the user wants Anthropic Claude / Cerebras / Whisper / a specific TTS voice. | ~800 ms turn | $-$$$$ |

Default pick: OpenAIRealtime2. Switch only if the user explicitly asks for custom voice/LLM, lower cost, or a non-OpenAI provider.

Quick start

Python

import asyncio
from getpatter import Patter, Twilio, OpenAIRealtime2

async def main():
    phone = Patter(carrier=Twilio(), phone_number="+15550001234")
    agent = phone.agent(
        engine=OpenAIRealtime2(),
        system_prompt=(
            "You are Mia, the AI receptionist for Acme Plumbing. "
            "Greet the caller warmly. Help them book a service visit. "
            "Keep replies under two sentences."
        ),
        first_message="Hi, this is Mia at Acme Plumbing — how can I help?",
    )
    await phone.serve(agent, tunnel=True)

asyncio.run(main())

TypeScript

import { Patter, Twilio, OpenAIRealtime2 } from "getpatter";

const phone = new Patter({
  carrier: new Twilio(),
  phoneNumber: "+15550001234",
});

const agent = phone.agent({
  engine: new OpenAIRealtime2(),
  systemPrompt:
    "You are Mia, the AI receptionist for Acme Plumbing. " +
    "Greet the caller warmly. Help them book a service visit. " +
    "Keep replies under two sentences.",
  firstMessage: "Hi, this is Mia at Acme Plumbing — how can I help?",
});

await phone.serve({ agent, tunnel: true });

Both expose a tunnel URL on startup (tunnel ready: https://abc.trycloudflare.com). Point your Twilio number's voice webhook at that URL (see configure-telephony), call the number, talk to Mia.

Pick the right mode

The default works for 80% of agents. Switch when:

| Want this | Pick this mode | Reference | |---|---|---| | Lowest possible latency, OpenAI voice | OpenAIRealtime2 (default) | [references/realtime-mode.md](references/realtime-mode.md) | | Better long-pause handling, ElevenLabs voice | ElevenLabsConvAI | [references/convai-mode.md](references/convai-mode.md) | | Mix custom STT / LLM / TTS (Anthropic, Deepgram, Cartesia, …) | Pipeline | [references/pipeline-mode.md](references/pipeline-mode.md) | | Save cost on high-volume outbound | Pipeline with Cerebras LLM + Deepgram STT + ElevenLabs Turbo | [references/pipeline-mode.md](references/pipeline-mode.md) | | Noisy line / speakerphone (denoiser, high-pass, AGC, semantic turn detection) | Pipeline audio levers | [references/pipeline-mode.md](references/pipeline-mode.md) |

Read the corresponding reference only when the user picks that mode — do not preload all three.

Outbound calls

The same Patter instance places outbound calls — but phone.call() always needs a running server to dial through (raises PatterConnectionError otherwise), and you choose how the call's lifecycle is reported:

• wait=True (completion-aware, since 0.6.3) — call() blocks until the

callee hangs up and returns a CallResult with the outcome, duration, transcript, and cost. This is what you want for scripts and campaigns: one await, one resolved call.

• wait=False (default, fire-and-forget) — call() returns at the moment

the carrier dials (not at hangup) and yields None/void. The call then lives entirely inside the running server — so something has to keep that server alive (a long-running serve(), or the async with / await using block below) for the conversation to continue.

The completion-aware form is the recommended pattern. async with Patter(...) (Python) / await using (TypeScript) boots the local server, keeps it alive for the call's lifetime, and tears it down cleanly on exit — no dangling process, no manual serve() task to cancel:

import asyncio
from getpatter import Patter, Twilio, OpenAIRealtime2

async def main():
    # `async with` runs the local server for the duration of the block.
    async with Patter(carrier=Twilio(), phone_number="+15550001234") as phone:
        agent = phone.agent(
            engine=OpenAIRealtime2(),
            system_prompt="You are Mia from Acme. Keep replies under two sentences.",
            first_message="Hi, this is Mia from Acme.",
        )
        result = await phone.call(
            to="+14155551234",
            agent=agent,
            voicemail_message="Sorry we missed you. Call back at +1...",
            wait=True,                 # block until the call ends → CallResult
        )
        # CallResult is frozen: call_id, outcome, status,
        # duration_seconds, transcript, cost, metrics.
        print(result.outcome)          # answered | voicemail | no_answer | busy | failed
        print(f"{result.duration_seconds:.0f}s · ${result.cost.total_usd:.4f}")

asyncio.run(main())

import { Patter, Twilio, OpenAIRealtime2 } from "getpatter";

// `await using` runs the local server, then disposes it when the block exits.
await using phone = new Patter({
  carrier: new Twilio(),
  phoneNumber: "+15550001234",
});

const agent = phone.agent({
  engine: new OpenAIRealtime2(),
  systemPrompt: "You are Mia from Acme. Keep replies under two sentences.",
  firstMessage: "Hi, this is Mia from Acme.",
});

const result = await phone.call({
  to: "+14155551234",
  agent,
  voicemailMessage: "Sorry we missed you. Call back at +1...",
  wait: true,                          // block until the call ends → CallResult
});
// CallResult is readonly: callId, outcome, status,
// durationSeconds, transcript, cost, metrics.
console.log(result.outcome);           // answered | voicemail | no_answer | busy | failed
console.log(`${result.durationSeconds}s · $${result.cost.totalUsd}`);

wait=True is timeout-bounded on ring_timeout (default 25 s), so a number that never picks up resolves to outcome="no_answer" rather than hanging forever. machine_detection is on by default since 0.6.3: Patter auto-detects voicemail, plays the call's voicemail_message before hanging up, and the CallResult comes back with outcome="voicemail". A non-empty voicemail_message implicitly enables AMD even if you pass machine_detection=False. Otherwise the live person hears first_message and the agent starts the conversation. See configure-telephony for the full set of outbound options.

System prompt — what matters for voice

Voice prompts are different from chat prompts. Patter doesn't add hidden instructions, so be explicit:

1. Identity: "You are , calling on behalf of ."
2. Goal: "Help the caller book a service visit." (one line)
3. Length rule: "Keep every reply under two sentences." (LLMs over-explain on voice.)
4. Stop conditions: "If the caller wants a human, call transfer_call."
5. Persona: "Warm, professional, never robotic. Say 'mm-hm' when listening."

The first_message is what the agent says immediately on pickup — keep it to 1 sentence under 8 seconds of audio.

Gotchas

• Patter has no Patter Cloud as of 0.7.0 — Patter(api_key=...) raises

NotImplementedError. Always use carrier=... + phone_number=....

• Built-in tools are always present: every agent gets transfer_call and

end_call for free. You don't have to register them.

• Mulaw vs PCM audio: Twilio carrier = mulaw 8 kHz, Telnyx = PCM 16 kHz.

Patter transcodes both transparently — you never touch raw audio unless you write a pipeline hook.

• tunnel=True is dev-only. For production, use a static webhook URL

(your own subdomain or paid ngrok). Cloudflare quick tunnels work but occasionally have a ~3 s WSS upgrade race on first call.

• first_message is played before the LLM responds. If you omit it,

there's an awkward 1–2 s pause while the LLM warms up.

• Barge-in works by default since 0.6.3 (VAD activation 0.8, deactivation

0.65 — tuned for room noise). If your callers are interrupting at the wrong moments, see [inspect-calls-and-metrics](../inspect-calls-and-metrics/) to read the VAD events from the call log.

• Callers cut off mid-pause? In Pipeline mode add a semantic

turn_detector (smart-turn or NAMO) and, on noisy lines, the denoiser / high_pass_hz / agc levers — see [references/pipeline-mode.md](references/pipeline-mode.md). In Realtime mode use openai_realtime_noise_reduction="far_field" and realtime_turn_detection — see [references/realtime-mode.md](references/realtime-mode.md).

• prewarm_first_message=False is the default since 0.6.3 — it was briefly

flipped to True mid-release and reverted because it conflicted with barge-in.

Common errors

| Symptom | Fix | |---|---| | RuntimeError: Patter Cloud is not implemented | Don't pass api_key=. Use carrier=Twilio() + phone_number="...". | | PatterConnectionError on call(..., wait=True) | No running server to dial through. Wrap the call in async with Patter(...) / await using, or place it while a serve() task is alive. | | Agent says nothing on pickup | Missing first_message=. Add one. | | LLM responses are 4 paragraphs long | Add "Keep replies under two sentences" to system prompt. | | Caller hears garbled audio | Wrong audio rate. Check you're using the right carrier (mulaw 8 kHz Twilio, PCM 16 kHz Telnyx). Patter handles this; user code only breaks it via custom pipeline hooks. | | Connection drops after 30 s | Either the tunnel died (use static URL) or barge-in hung. Check MetricsStore for the last event. |

Related skills

• [setup-patter](../setup-patter/) — install + env vars (run this first if user hasn't).
• [configure-telephony](../configure-telephony/) — set up the Twilio/Telnyx webhook.
• [add-tools-and-handoffs](../add-tools-and-handoffs/) — custom tools, transfer_call, guardrails.
• [inspect-calls-and-metrics](../inspect-calls-and-metrics/) — live dashboard, cost per call, transcript.

References

• Patter overview: ·
• Concepts:
• Agent configuration: ·
• Examples:

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: PatterAI
• Source: PatterAI/skills
• License: MIT
• Homepage: https://www.skills.sh/patterai/skills

Install and usage instructions live in the source repository linked above.