Realtime

Let one realtime model hear the caller, answer, and use your tools. The model takes audio directly. turn_detection chooses who ends the caller’s turn, and the model’s voice speaks the reply by default in this example. On this page:

Quickstart - a complete realtime agent
Pros and cons - strengths and limits
Bind the model - choose the voice
Choose turn detection - defaults and limits
Attach tools - extend the same desk
Advanced - a separate synthesizer
Troubleshooting - fix voice and turn problems
Where to go next - examples and references

Quickstart

Create a new package with a LiveKit target. If initialization opens the console, select LiveKit and finish creating the package.

Terminal

unmute init voice-desk

Replace voice-desk/agent.yaml with this complete file:

voice-desk/agent.yaml

version: 1
name: voice-desk
architecture: realtime
entry_agent: desk
secrets:
  - OPENAI_API_KEY
models:
  realtime:
    - name: voice
      provider: openai
      model: gpt-realtime
      voice: marin
      turn_detection: semantic
agents:
  desk:
    instructions: instructions.md
    realtime: voice
channels:
  web:
    kind: realtime_audio
capacity:
  peak_sessions: 1
  max_sessions: 2
  avg_session_duration: 3m

Keep the scaffold’s targets.yaml, but remove its target-level models: overrides because the model palette above replaces the scaffold’s. Keep its framework provider and version pin, and remove any phone connection: for this browser example. Replace voice-desk/instructions.md with this complete prompt:

voice-desk/instructions.md

You are a friendly information desk. Answer in short spoken sentences.
Ask one question at a time. If you do not know an answer, say so.

Set OPENAI_API_KEY in your shell or the package’s .env, then run:

Terminal

unmute validate voice-desk
unmute compile voice-desk
unmute dev voice-desk --target livekit

Use Docker for this LiveKit run. The Pipecat target is another option; it runs locally with uv. The example below stays with this same desk agent.

Pros and cons

Pick Realtime when direct audio interaction matters and the call fits one agent with tools. It keeps more control over turn detection and voice than Unmute’s Live architecture.

Pros	Cons
Direct audio input retains clues that may be lost in transcription	Audio understanding and reasoning stay with the chosen realtime model
Fewer stage boundaries can reduce response delay	Provider, turn settings, network, and tools still determine the actual wait
Choose a turn mode and optionally keep a separate synthesizer	A separate synthesizer adds another component and its response time
Tools and knowledge searches remain available	Unmute currently supports no tasks, handoffs, saved variables, tracing, or phone route here

For a workflow that collects, confirms, and acts across several tasks, use Cascade. Its extra control can be worth a longer pause. See the comparison and further reading for the wider tradeoff.

1. Bind the realtime model

The desk agent names the voice entry through realtime:. Its OpenAI model listens and speaks, so this package has no separate transcriber or reasoning binding. An unused models.think entry may remain, but the agent cannot bind it with think:. Remove unused entries to keep the package clear.

cascade | realtime | live

required

Set realtime explicitly. Omitted means cascade.

a models.realtime entry name

required

Agent-level model binding. Replaces the agent’s think: and its normal speak: binding.

string

required

Name of the realtime model entry. The agent refers to it by this name.

openai

required

Supported realtime provider on Pipecat and LiveKit. The SLNG target cannot run this architecture.

string

required

Model ID forwarded to the provider. Validation does not check account access or model availability.

string

Voice on the realtime entry. Supply this or an agent-level speak: binding, never both. Omitting both is refused.

string

Optional author note. It does not change the runtime.

2. Choose who ends the turn

The quickstart writes semantic to make the choice explicit on both targets. Change only that entry’s turn_detection when comparing the options.

server_vad | semantic | local

Optional turn choice on the realtime entry. An omitted value leaves the integration’s default in place; the targets can choose differently.

Value	What decides
`server_vad`	The provider’s silence detector
`semantic`	The provider’s detector judging whether the caller finished their thought
`local`	The framework’s local turn handling
Omitted	Pipecat leaves the provider default; the LiveKit plugin supplies its default

The current integrations default to server silence detection on Pipecat and semantic detection on LiveKit. Write an explicit value when comparing the two frameworks. Do not add models.turn, package-level turn:, or listen: to a realtime package, even with local. Those cascade sections are refused. conversation.interruption.minimum_words requires turn_detection: local because it gates a local turn start.

3. Add tools and knowledge

Keep the same tool files and attachments described in Add a local tool. Merge the tool name into the package’s tools: list and agents.desk.tools, keeping the existing instructions and realtime binding. The realtime model requests tool calls; the generated application runs them and returns the results. It has no separate backend: model. For document questions, add a knowledge search tool in the same way. The pharmacy example combines local tools, knowledge, and semantic turn detection. After each change, stop the old run and check the same desk:

Terminal

unmute validate voice-desk
unmute dev voice-desk --target livekit

Ask a question that requires the tool. Pause in the middle of a sentence, then finish it. Check both the spoken reply and the tool row in the dev page.

Advanced

Use a separate speech synthesizer

The realtime model can return text while a synthesizer speaks it. This is sometimes called a half cascade.Remove voice: from the realtime entry named voice. Merge the following entries into the quickstart, keeping its existing realtime model and agent binding:

voice-desk/agent.yaml

secrets:
  - SLNG_API_KEY
models:
  speak:
    spoken_voice:
      provider: slng
      model: "deepgram/aura:2"
      voice: aura-2-thalia-en
agents:
  desk:
    speak: spoken_voice

Append the secret name rather than replacing OPENAI_API_KEY, and set its value in your environment. Validate and compile both declared targets before starting another call. Choose other supported synthesizers from Voices.

Switch architecture or extend the workflow

Follow Switch architecture to replace the models and bindings together. The Realtime API can accept instruction updates during a call. Unmute currently supports one agent, browser audio, tools, and knowledge; it does not yet provide task and handoff transitions consistently across both code targets. Tasks, task groups, handoffs, variables, pre-fetch, tracing, MCP tools, and phone connections are unavailable. Use cascade if your workflow needs them.

Troubleshooting

Validation says both voice and speak are set

The realtime entry and agent both name a speaker. Fix: remove the entry’s voice: for a separate synthesizer, or remove the agent’s speak: to use the model’s voice.

Validation says there is no voice

Neither the realtime entry nor the agent names a speaker. Fix: add voice: marin to the OpenAI realtime entry, or configure the Advanced synthesizer setup.

The agent cuts in while I spell a reference

A silence detector may treat the pause as the end of the turn. Fix: try turn_detection: semantic, then retest the same phrase. Explain expected pauses in the prompt if needed.

Local turn detection still rejects models.turn

local selects framework behavior; it does not enable the cascade model sections. Fix: remove models.turn and its top-level selector. Keep the choice on the realtime entry.

Get started

Build the agent

Best practices

Run and observe

Optimization

Deploy

Phone calls

Targets and models

Configuration files

CLI

Community

Releases

Quickstart

Pros and cons

1. Bind the realtime model

2. Choose who ends the turn

3. Add tools and knowledge

Advanced

Troubleshooting

Validation says both voice and speak are set

Validation says there is no voice

The agent cuts in while I spell a reference

Local turn detection still rejects models.turn

Where to go next

Live

Knowledge bases

​Quickstart

​Pros and cons

​1. Bind the realtime model

​2. Choose who ends the turn

​3. Add tools and knowledge

​Advanced

​Troubleshooting

​Validation says both voice and speak are set

​Validation says there is no voice

​The agent cuts in while I spell a reference

​Local turn detection still rejects models.turn

​Where to go next

Live

Knowledge bases

Quickstart

Pros and cons

1. Bind the realtime model

2. Choose who ends the turn

3. Add tools and knowledge

Advanced

Troubleshooting

Validation says both voice and speak are set

Validation says there is no voice

The agent cuts in while I spell a reference

Local turn detection still rejects models.turn

Where to go next