> ## Documentation Index
> Fetch the complete documentation index at: https://unmute.ai/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Unmute compiles to exactly three targets. Pipecat and LiveKit are code targets: compile writes a Python project you run. SLNG is a hosted target: compile writes a deployment body and SLNG runs the agent, so it has no `unmute dev`. Those three are the only values `provider` accepts in `targets.yaml`. Deepgram and ElevenLabs appear in these docs as model vendors, which is not the same thing as a target, and `slng` is both.
> The Go structs in `internal/spec` and `internal/ir` are the schema truth. Check a field against them, or run `unmute validate`, rather than against what you remember.
# Realtime
> Run one model for spoken conversation, then choose turn detection, tools, and an optional separate voice.
Let one realtime model hear the caller, answer, and use your tools.
The model takes audio directly. `turn_detection` chooses who ends the caller's turn, and the model's voice speaks the reply by default in this example.
On this page:
* [Quickstart](#quickstart) - a complete realtime agent
* [Pros and cons](#pros-and-cons) - strengths and limits
* [Bind the model](#1-bind-the-realtime-model) - choose the voice
* [Choose turn detection](#2-choose-who-ends-the-turn) - defaults and limits
* [Attach tools](#3-add-tools-and-knowledge) - extend the same desk
* [Advanced](#advanced) - a separate synthesizer
* [Troubleshooting](#troubleshooting) - fix voice and turn problems
* [Where to go next](#where-to-go-next) - examples and references
## Quickstart
Create a new package with a LiveKit target. If initialization opens the console, select LiveKit and finish creating the package.
```sh Terminal theme={null}
unmute init voice-desk
```
Replace `voice-desk/agent.yaml` with this **complete file**:
```yaml voice-desk/agent.yaml theme={null}
version: 1
name: voice-desk
architecture: realtime
entry_agent: desk
secrets:
- OPENAI_API_KEY
models:
realtime:
- name: voice
provider: openai
model: gpt-realtime
voice: marin
turn_detection: semantic
agents:
desk:
instructions: instructions.md
realtime: voice
channels:
web:
kind: realtime_audio
capacity:
peak_sessions: 1
max_sessions: 2
avg_session_duration: 3m
```
Keep the scaffold's `targets.yaml`, but remove its target-level `models:` overrides because the model palette above replaces the scaffold's.
Keep its framework provider and version pin, and remove any phone `connection:` for this browser example. Replace `voice-desk/instructions.md` with this complete prompt:
```markdown voice-desk/instructions.md theme={null}
You are a friendly information desk. Answer in short spoken sentences.
Ask one question at a time. If you do not know an answer, say so.
```
Set `OPENAI_API_KEY` in your shell or the package's `.env`, then run:
```sh Terminal theme={null}
unmute validate voice-desk
unmute compile voice-desk
unmute dev voice-desk --target livekit
```
Use Docker for this LiveKit run. The [Pipecat target](/targets/pipecat) is another option; it runs locally with `uv`.
The example below stays with this same `desk` agent.
## Pros and cons
Pick Realtime when direct audio interaction matters and the call fits one
agent with tools. It keeps more control over turn detection and voice than
Unmute's Live architecture.
| Pros | Cons |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- |
| Direct audio input retains clues that may be lost in transcription | Audio understanding and reasoning stay with the chosen realtime model |
| Fewer stage boundaries can reduce response delay | Provider, turn settings, network, and tools still determine the actual wait |
| Choose a turn mode and optionally keep a separate synthesizer | A separate synthesizer adds another component and its response time |
| Tools and knowledge searches remain available | Unmute currently supports no tasks, handoffs, saved variables, tracing, or phone route here |
For a workflow that collects, confirms, and acts across several tasks, use
[Cascade](/build/architecture/cascade#3-add-tools-tasks-or-saved-state).
Its extra control can be worth a longer pause. See the
[comparison and further reading](/build/architecture/overview#pros-and-cons)
for the wider tradeoff.
## 1. Bind the realtime model
The `desk` agent names the `voice` entry through `realtime:`.
Its OpenAI model listens and speaks, so this package has no separate transcriber or reasoning binding.
An unused `models.think` entry may remain, but the agent cannot bind it with `think:`. Remove unused entries to keep the package clear.
Set `realtime` explicitly. Omitted means `cascade`.
Agent-level model binding. Replaces the agent's `think:` and its normal `speak:` binding.
Name of the realtime model entry. The agent refers to it by this name.
Supported realtime provider on Pipecat and LiveKit. The SLNG target cannot run this architecture.
Model ID forwarded to the provider. Validation does not check account access or model availability.
Voice on the realtime entry. Supply this or an agent-level `speak:` binding, never both. Omitting both is refused.
Optional author note. It does not change the runtime.
## 2. Choose who ends the turn
The quickstart writes `semantic` to make the choice explicit on both targets.
Change only that entry's `turn_detection` when comparing the options.
Optional turn choice on the realtime entry. An omitted value leaves the integration's default in place; the targets can choose differently.
| Value | What decides |
| ------------ | ---------------------------------------------------------------------------- |
| `server_vad` | The provider's silence detector |
| `semantic` | The provider's detector judging whether the caller finished their thought |
| `local` | The framework's local turn handling |
| Omitted | Pipecat leaves the provider default; the LiveKit plugin supplies its default |
The current integrations default to server silence detection on Pipecat and semantic detection on LiveKit.
Write an explicit value when comparing the two frameworks.
Do not add `models.turn`, package-level `turn:`, or `listen:` to a realtime package, even with `local`.
Those cascade sections are refused. `conversation.interruption.minimum_words` requires `turn_detection: local` because it gates a local turn start.
## 3. Add tools and knowledge
Keep the same tool files and attachments described in [Add a local tool](/build/tools/python).
Merge the tool name into the package's `tools:` list and `agents.desk.tools`, keeping the existing `instructions` and `realtime` binding.
The realtime model requests tool calls; the generated application runs them and returns the results.
It has no separate `backend:` model.
For document questions, add a [knowledge search tool](/build/tools/knowledge) in the same way.
The [pharmacy example](https://github.com/slng-ai/unmute/tree/main/examples/pharmacy-refills) combines local tools, knowledge, and semantic turn detection.
After each change, stop the old run and check the same desk:
```sh Terminal theme={null}
unmute validate voice-desk
unmute dev voice-desk --target livekit
```
Ask a question that requires the tool. Pause in the middle of a sentence, then finish it.
Check both the spoken reply and the tool row in the dev page.
## Advanced
The realtime model can return text while a synthesizer speaks it. This is sometimes called a half cascade.
Remove `voice:` from the realtime entry named `voice`. Merge the following entries into the quickstart, keeping its existing realtime model and agent binding:
```yaml voice-desk/agent.yaml theme={null}
secrets:
- SLNG_API_KEY
models:
speak:
spoken_voice:
provider: slng
model: "deepgram/aura:2"
voice: aura-2-thalia-en
agents:
desk:
speak: spoken_voice
```
Append the secret name rather than replacing `OPENAI_API_KEY`, and set its value in your environment.
Validate and compile both declared targets before starting another call.
Choose other supported synthesizers from [Voices](/models/tts).
Follow [Switch architecture](/build/architecture/overview#3-replace-the-models-and-agent-bindings) to replace the models and bindings together.
The Realtime API can accept instruction updates during a call.
Unmute currently supports one agent, browser audio, tools, and knowledge; it does not yet provide task and handoff transitions consistently across both code targets.
Tasks, task groups, handoffs, variables, pre-fetch, tracing, MCP tools, and phone connections are unavailable.
Use cascade if your workflow needs them.
## Troubleshooting
### Validation says both voice and speak are set
The realtime entry and agent both name a speaker.
**Fix:** remove the entry's `voice:` for a separate synthesizer, or remove the agent's `speak:` to use the model's voice.
### Validation says there is no voice
Neither the realtime entry nor the agent names a speaker.
**Fix:** add `voice: marin` to the OpenAI realtime entry, or configure the Advanced synthesizer setup.
### The agent cuts in while I spell a reference
A silence detector may treat the pause as the end of the turn.
**Fix:** try `turn_detection: semantic`, then retest the same phrase. Explain expected pauses in the prompt if needed.
### Local turn detection still rejects models.turn
`local` selects framework behavior; it does not enable the cascade model sections.
**Fix:** remove `models.turn` and its top-level selector. Keep the choice on the realtime entry.
## Where to go next
Use a voice model with a reasoning backend.
Answer questions from documents.