> ## Documentation Index
> Fetch the complete documentation index at: https://unmute.ai/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Unmute compiles to exactly three targets. Pipecat and LiveKit are code targets: compile writes a Python project you run. SLNG is a hosted target: compile writes a deployment body and SLNG runs the agent, so it has no `unmute dev`. Those three are the only values `provider` accepts in `targets.yaml`. Deepgram and ElevenLabs appear in these docs as model vendors, which is not the same thing as a target, and `slng` is both.
> The Go structs in `internal/spec` and `internal/ir` are the schema truth. Check a field against them, or run `unmute validate`, rather than against what you remember.
# Architecture
> Choose a pipeline, switch an existing agent, and check that its models and bindings agree.
Choose how your agent listens, reasons, and speaks with `architecture:` in `agent.yaml`.
The architecture chooses the pipeline; `targets.yaml` chooses the framework that runs it.
Changing one does not change the other.
On this page:
* [Quickstart](#quickstart) - run a complete example
* [Pros and cons](#pros-and-cons) - control and response time
* [Choose the pipeline](#1-choose-the-pipeline) - compare the three choices
* [Check your package](#2-check-what-your-package-needs) - keep required capabilities
* [Replace the bindings](#3-replace-the-models-and-agent-bindings) - switch the same agent
* [Validate and run](#4-validate-compile-and-start-a-new-call) - check every target
* [What compiles where](#what-compiles-where) - target support
* [Advanced](#advanced) - voices and turn settings
* [Troubleshooting](#troubleshooting) - fix a refused switch
* [Further reading](#further-reading) - LiveKit and Coval comparisons
* [Where to go next](#where-to-go-next) - build each shape
## Quickstart
From a clone of [Unmute](https://github.com/slng-ai/unmute), run the complete realtime example.
Install the [CLI and runtime prerequisites](/start/installation) first, and set `OPENAI_API_KEY` in your shell or existing repository-root `.env`.
```sh Repository root theme={null}
unmute validate examples/pharmacy-refills
unmute compile examples/pharmacy-refills
unmute dev examples/pharmacy-refills --target pipecat
```
Ask for a refill on the demo reference RX4821B, pausing halfway through the reference.
Stop the process before trying `--target livekit` with the same package.
Pipecat runs locally with `uv`; LiveKit uses Docker Compose.
That is the loop: **choose the architecture, bind its models, validate, talk**.
The steps below explain how to switch an existing package.
## Pros and cons
| Architecture | Pros | Cons |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| [Cascade](/build/architecture/cascade) | Independent model choices and clearer stage-by-stage diagnosis; Unmute's richest workflow controls on LiveKit and Pipecat | More components to tune and usually more latency; reasoning sees a transcript rather than the caller's tone |
| [Realtime](/build/architecture/realtime) | Direct audio understanding, potentially quicker replies, a choice of turn mode and optional separate voice | Less freedom to change the reasoning model; Unmute currently limits it to one agent without tasks or saved state |
| [Live](/build/architecture/live) | The model handles turns, with a separate backend for tool work | No authored turn controls or separate synthesizer; Unmute currently limits it to one agent without tasks or saved state |
**The extra control is often worth the latency.** If a call must collect details,
confirm them, run an action, and recover when it fails, choose the architecture
that lets you express those steps clearly. A faster first reply is only one
part of a successful call.
These are current Unmute capabilities, not universal limits of speech-to-speech APIs.
Compare completed actions, recovery from failures, and response time before choosing.
## 1. Choose the pipeline
| Value | Who does the work | Choose it when |
| ---------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| `cascade` | Separate transcription, reasoning, speech synthesis, and turn detection | You need tasks, handoffs, saved state, or phone routes |
| `realtime` | One speech-to-speech model, with a choice of turn detection and voice output | You want direct audio with explicit turn settings |
| `live` | A voice model with an optional reasoning backend | You want the model to manage conversation while its backend runs tools |
Package-level pipeline choice. Omitted means `cascade`. It applies to the whole package, not to one agent or target.
Both speech-to-speech architectures support Pipecat and LiveKit browser audio.
The SLNG target supports `cascade` only. Model providers and target frameworks are separate choices.
## 2. Check what your package needs
Before switching a cascade, check this list. Removing a feature changes what your agent can do.
Keep `cascade` if any required feature is unavailable on the destination architecture.
| Capability | Cascade | Realtime | Live |
| --------------------------------------------- | ------------------------ | -------- | ---------------------- |
| Local tools and knowledge lookup | Yes | Yes | Yes, through a backend |
| Multiple agents, tasks, task groups, handoffs | LiveKit and Pipecat | No | No |
| Variables and pre-fetch | Yes | No | No |
| Tracing and MCP tools | Yes on supported targets | No | No |
| Phone routes and human transfers | On supported routes | No | No |
| Separate speech synthesizer | Yes | Optional | No |
For example, `salon-concierge` uses tasks, handoffs, variables, and tracing.
Changing its architecture line cannot preserve that workflow. Start with a single-agent S2S example instead.
## 3. Replace the models and agent bindings
Switch a simple agent called `desk` by replacing its `architecture:`, entire `models:` block, and model bindings.
These are **replacement fragments**, not complete packages. Keep its `instructions`, tool attachments, and other supported settings.
Do not paste a second `models:` or `agents:` block alongside the first.
```yaml Realtime: agent.yaml theme={null}
architecture: realtime
models:
realtime:
- name: voice
provider: openai
model: gpt-realtime
voice: marin
turn_detection: semantic
agents:
desk:
instructions: instructions.md
realtime: voice
```
```yaml Live: agent.yaml theme={null}
architecture: live
models:
live:
- name: voice
provider: openai
model: gpt-live-1
voice: marin
backend: reasoning
think:
reasoning:
provider: openai
model: gpt-5.6-terra
agents:
desk:
instructions: instructions.md
live: voice
```
Use your existing agent name in place of `desk`, and keep `entry_agent` pointing to it.
The [cascade guide](/build/architecture/cascade) supplies the corresponding listening, reasoning, and speaking bindings for switching back.
| Destination | Remove or replace |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Realtime | Remove `live:` or agent `think:`/`speak:` bindings; replace the live or cascade models with `models.realtime` |
| Live | Remove `realtime:` or agent `think:`/`speak:` bindings; replace their models with `models.live` and the backend's `models.think` entry |
| Cascade | Remove agent `live:`/`realtime:` and their model sections; restore `models.listen`, `models.think`, `models.speak`, `models.turn`, and agent `think:`/`speak:` |
For either S2S destination, remove package-level `listen:` and `turn:` selectors and their model sections.
For live, also remove `conversation.interruption`; the model owns interruption behavior.
Realtime can retain a separate synthesizer only through the [Advanced voice setup](/build/architecture/realtime#advanced).
Update `secrets:` to match the remaining providers. The S2S fragments above require `OPENAI_API_KEY`.
In `targets.yaml`, remove model overrides for entries you deleted. Keep target names, providers, and supported framework pins.
Remove phone connections and keep the browser channel when switching to S2S.
## 4. Validate, compile, and start a new call
Run these from the directory containing your package. `voice-desk` is its directory name here.
```sh Terminal theme={null}
unmute validate voice-desk
unmute compile voice-desk
unmute dev voice-desk --target livekit
```
Without `--target`, validate and compile check every declared target.
Use `--target pipecat` to run that framework if the package declares it.
Stop the old dev process and start a new call after switching; an active session keeps its current architecture.
Test a greeting, a tool call, knowledge lookup, and interruption.
A successful compile checks the package; it does not prove provider access or audible behavior.
## What compiles where
| Architecture | LiveKit | Pipecat | SLNG |
| ------------ | ------- | ------- | ---- |
| `cascade` | yes | yes | yes |
| `realtime` | yes | yes | no |
| `live` | yes | yes | no |
S2S support covers browser audio. Check the feature limits in step 2 before switching.
## Advanced
Realtime supports `turn_detection` and an optional `speak:` binding.
Follow [Realtime](/build/architecture/realtime#advanced) to add them to the same agent.
Live decides turns itself and uses its own voice.
Keep the prompt and local tools as similar as the architectures allow.
Compare the dev page's reported measurements and the conversation you hear.
Speech-to-speech removes separate transcription and synthesis stages, but does not guarantee a faster call.
See [Latency](/optimization/latency).
## Troubleshooting
### Validation says a model section is not used
The architecture changed, but an old model section or selector remains.
**Fix:** replace the models and bindings together using step 3, then validate again.
### A target overrides a model that no longer exists
`targets.yaml` still names a deleted cascade or backend entry.
**Fix:** delete that override or point it at a supported remaining model. Do not add a dummy model just to satisfy it.
### Validation refuses tasks or saved variables
The destination S2S architecture does not support those capabilities.
**Fix:** keep `architecture: cascade`, or start a separate single-agent package with the supported workflow.
### The call still uses the old architecture
The running process was started from an earlier build.
**Fix:** stop it, rerun `unmute dev`, and open a new call.
## Further reading
[LiveKit's pipeline and realtime comparison](https://livekit.com/blog/realtime-vs-cascade)
explains the balance between latency, direct audio understanding, and modular
control. It also describes how streaming and hybrid designs narrow the gap.
[Coval's speech-to-speech and cascade comparison](https://www.coval.ai/blog/speech-to-speech-vs-cascaded-voice-ai-which-architecture-should-you-deploy/)
focuses on workflow control, diagnosing failures, and evaluating completed
conversations. Use the tables above for what Unmute supports today.
## Where to go next
Build with separate models.
Choose the turn and voice.
Add a voice model and backend.