How it works
- The LiveKit or Pipecat adapter calls the
start_*andend_*helpers on session events. Telemetryowns one TracerProvider for the call. It stamps the call’s identity on every span.- Each sink adds a batch processor and an OTLP/HTTP exporter to that provider.
- A sink with
filter=MetadataOnly()sends no text. It keeps only an allow-list of attributes. close()ends the call span and flushes every sink.
Telemetry
Telemetry arguments
Sink
required
Where the spans go. Pass at least one.
str
required
The agent’s name. It is the trace name and the service name.
str
required
The call’s ID, like the room or session ID. It is stamped on every span as
session.id. An empty agent or call_id raises ValueError.Mapping[str, AttributeValue] | None
default:"None"
Extra resource attributes, like
deployment.environment.name.Sequence[SpanProcessor]
default:"()"
Extra span processors that run before the sinks, for example one that adds your own attributes.
Set[str]
default:"{\"openrouter\", \"slng\"}"
Providers that pass a request on to another provider named in the model id, like OpenRouter. A speech span for one of them records the provider behind it as
unmute.upstream.provider. For slng/fish/tts:s2.1-pro the value is fish. Add your own gateway here. The default is GATEWAYS.Call the helpers by hand
The adapters call these helpers for you. Call them yourself only on a framework the library has no adapter for.agent.py
Helpers
close waits up to 10 seconds by default (CLOSE_TIMEOUT_SECONDS) so a call that no simulation placed has time to be filed with Coval. It returns whether the flush finished.
Rules for the end helpers:
- Latencies are in seconds. A missing, negative or infinite latency is left out, never sent as 0.
error="<type>"setserror.typeand the span status to ERROR. On Pipecat, thellmspan gets the error type of anErrorFramefrom the LLM service.- An attribute with no value is not set.
gen_ai.provider.nameis lowercase, so one provider has one name on both frameworks.- A text attribute keeps up to 256 KiB, so a long prompt stays valid JSON.
describe_call(origin=..., framework=...) sets unmute.call.origin (phone, browser or websocket) and unmute.call.framework (livekit or pipecat) on the call span. Values given before start_call wait and are set when it starts. After the call span ended, it does nothing. The adapters stamp the framework themselves. correlate_coval sets the origin.
On a plain agent
On LiveKit,UnmuteAgent traces the session for you. trace_session is for a plain Agent:
Agent gets an llm span through traced_llm, which also fills {{placeholders}} from the trace’s state. tts spans need a tts_node on your agent:
ModelCallLLM(model, trace=trace) also gives an llm span when a ModelCall answers.
On Pipecat, leave Pipecat’s own enable_tracing off, or the call is traced twice. Set PipelineParams(enable_metrics=True, enable_usage_metrics=True), or the spans have no time to first byte and no tokens. TelemetryObserver logs one warning when these are not on.
TelemetryObserver
Telemetry
required
The call’s telemetry, with its call span started.
str
required
The model name the
llm spans carry. After a handoff gives the next agent its own model call, its spans name that one.str
required
Who serves the model, like
openai.UnmuteFlows | None
default:"None"
The call’s tasks, which tell each request’s prompt site.
ModelCallLLMService in flows, the observer also sets the service’s trace_headers. Each model request then carries a traceparent, so the router’s spans join the call’s trace. On Pipecat they are children of the turn span, not of the request’s llm span. The observer gets frames through a queue, so the llm span may open after the request is sent.
Sinks
Each of the first three has
from_env() and from_settings(settings). from_env raises TelemetryConfigError and names every missing variable. A missing Coval key only logs a warning. LangfuseSettings, LogfireSettings and CovalSettings read the same variables. Pass one to from_settings to load them at startup with check_settings, or to set values in code. Both methods take sink options too, like Langfuse.from_env(filter=MetadataOnly()). Settings lists every variable.
Sink arguments
str
required
Langfuse. The project’s public key.str
required
Langfuse. The project’s secret key.str
required
Langfuse. Like https://cloud.langfuse.com.str
required
Logfire. A write token.str | None
required
Coval. The Coval API key. None exports nothing.str | None
default:"None"
Coval. The simulation ID for a local run. An adapter’s correlate_coval reads it.bool
default:"True"
Coval. File a call no simulation placed as a Coval conversation at the end. False files nothing.str | None
default:"None"
Otlp. The full traces URL, like http://collector:4318/v1/traces. With none, the OpenTelemetry SDK reads its own OTEL_EXPORTER_OTLP_* variables.dict[str, str]
default:"{}"
Otlp. Extra HTTP headers, like an API key.MetadataOnly | None
default:"None"
Every sink. Lets only metadata through.
Datadog
Send the spans with anOtlp sink to an OpenTelemetry Collector. The Collector exports them to Datadog. The library needs no Datadog package.
collector.yaml
site to your Datadog site, like datadoghq.eu. Keep the API key in the Collector’s environment, not in the agent. Add filter=MetadataOnly() to the Otlp sink to send no text.
Coval
Coval needs to know which Coval run the call belongs to. Spans wait in memory, up to 1024, untilcoval.activate(simulation_id=...) or coval.activate(conversation_id=...). Pass method="dispatch_metadata" or similar to record how the ID was found. It becomes coval.correlation.method on the call span. activate raises ValueError unless you pass exactly one of the two IDs. It raises RuntimeError when no Telemetry was built with the sink, or the call already has a Coval ID. It sends the held spans on the calling thread, so call it with asyncio.to_thread from async code.
An adapter finds the ID for you. Call correlate_coval once, with the sink the call’s Telemetry was built with:
runner_args is Pipecat’s RunnerArguments. It also tells telemetry how the caller arrived, with describe_call(origin=...).
The ID is the
coval.simulation_id key, or any key that ends in x-coval-simulation-id, in any case. coval_simulation_id(mapping) does the lookup. On LiveKit the first ID wins, and the sink activates once on a worker thread. On Pipecat it activates at once, on the calling thread.
A call that no simulation placed is filed when it ends. A real call or a local run then shows up in Coval’s trace search.
Telemetry.close()flushes, then the provider shuts the Coval exporter down.- The exporter reads the transcript, the session ID, the agent name and the origin from the held call span. It merges consecutive lines of one speaker.
- It posts them to
https://api.coval.dev/v1/conversations:submitand waits at most 5 seconds. - With the conversation ID it sends every held span with the
X-Conversation-Idheader. The call span getscoval.conversation_idandcoval.correlation.method=conversation_submit.
log_no_simulation(log, call="call") says no simulation placed the call, so its trace is filed at the end. log_activated(log, simulation_id, method, call=None) says the sink is now bound to a simulation.
A failed submit logs a warning and never raises. The held spans are then dropped. A call is not filed, and the log says why, when:
- it already has a simulation or conversation ID,
- there is no Coval API key,
- the sink is
Coval(file_calls=False), - the sink has a
filter, because metadata only never sends the transcript, - the call span has no transcript.
Metadata only
- It keeps an attribute only if the key is on the list and the value is a number, a bool or a short safe string.
- It renames any span not on the list to
operation. - It drops events, link attributes and the status description.
- It keeps only allowed resource keys.
MetadataOnly arguments
frozenset[str]
default:"DEFAULT_ATTRIBUTES"
Span attribute keys to keep. Pass
DEFAULT_ATTRIBUTES | {"my.key"} to allow more.frozenset[str]
default:"DEFAULT_SPAN_NAMES"
Span names to keep.
frozenset[str]
default:"DEFAULT_RESOURCE"
Resource attribute keys to keep.
MetadataOnly is a frozen Pydantic dataclass. A value of the wrong type raises a ValidationError.
filter.apply(span) filters one span. filter.apply_all(spans) filters a batch. The sink uses apply_all on each export. It builds the filtered resource once for each resource the spans share, so a batch of a call’s spans costs less than one apply per span.
Write a sink
Sink is the plug point for a new provider. Subclass it and return an exporter:
filter when one is set. Build sinks once per call, with the call’s Telemetry.
Sink is a Pydantic dataclass. A stdlib @dataclass(kw_only=True) subclass, as above, works. So does a pydantic.dataclasses.dataclass(kw_only=True) subclass. With the Pydantic one, a field of the wrong type, like api_key=3, raises a ValidationError when you build the sink. The built-in sinks check their fields the same way.
Span contract
Both adapters emit these spans through the helpers. Do not create them by hand.conversationis the call. It starts a new trace. When it closes, its input is the transcript, one entry per line. Its output is how the call ended:{"end_reason", "state", "confirmed"}.turnis one caller message and everything the agent does until the next one. Its output is every line the agent said, joined by a space. Only a turn where the agent speaks first has no caller input.sttis one transcript.ttsis one piece of speech.llmis one model request: the prompt as filled, the tools offered, and the text and tool calls the model chose.llm_tool_callis one tool run.- An event span is one decision, like a task starting, a step skipped or a handoff. It has no length. Langfuse does not show OpenTelemetry span events, so each one is a span.
task_stepsits under the call, not around turns. One turn can finish a task and start the next.
session.id, gen_ai.conversation.id (both the call ID) and agent.name. The resource has service.name set to the agent name.
Attributes of each span
Attributes of each span
Where each adapter reads each span
Where each adapter reads each span
On LiveKit, STT is read from the caller’s message, not from
stt_node. Overriding stt_node would stop LiveKit reusing the STT stream each time a step starts. A subclass that overrides tts_node loses the tts spans unless it calls super().tts_node(...). LiveKit’s own spans stay off, and the adapter does not call set_tracer_provider.What a tool span shows
A tool that takes no parameters shows{} as its input. A tool span’s output is what the tool gave back, in three cases that need care:
- A task call’s span lasts the whole task. Its output is the task result the main agent reads, not Pipecat’s first “running” result.
- A step’s closing tool shows what it really returned, like the customer record. The model reads only how the step ended, and the
task_stepspan shows that. - A handoff shows
{"status": "handed off"}on both frameworks. A step that hands off ends itstask_stepspan with the outcomehanded_off.
Model requests that make no reply
Three kinds of model request make no reply. All are stillllm spans in the turn they belong to.
- A step’s finish check. Its tools are only
finishandnot_yet, and a call is required. Anot_yetis an answer, never run, so it has no tool span. Afinishit chose runs as a tool, under the call id the check gave it. - The decision on a held save, made when the call ends. It sits in the last turn. LiveKit settles before it ends the turn, and a Pipecat host calls
flows.settle()before theEndFrame. Its calls are answers too, never run. - A summary, for a handoff or a step with
history: summary. Its site issummary, it offers no tools, and its output is the summary text.
When a turn starts and ends
A turn belongs to the caller until the agent says a line, starts speech or calls a tool. A model request alone does not count. A request cut off because the caller went on speaking leaves the rest of their words in the same turn. More caller input before that joins the open turn, like a caller who pauses mid-sentence. Caller input after that starts the next turn.Following the turns in your own adapter
TurnTracker. It numbers the turns and keeps the open turn span. You decide when a turn starts and what its text and latencies are.
method
start().
Opens the next turn under the call and returns its span. End the earlier turn first.method
open(answering=True).
Returns the open turn, or opens one when none is. It marks the turn answered unless you pass answering=False.bool
True while a turn is open and nothing the agent did has answered in it.
method
end(**fields).
Ends the open turn through Telemetry.end_turn. It does nothing when no turn is open.TypedDict
Names the fields of
end: user_text, agent_lines, interrupted, e2e, stt_final, eou, llm_ttft, tts_ttfb and error. They are the same as the arguments of end_turn.Events
The events are the task stages (task.started, task.completed, task.refused_busy, group.step_skipped, task.handed_off and the rest of TaskStage), handoff with {"from", "to"}, and the idle nudges (idle_nudges.armed, idle_nudges.first.fired and the rest). A nudge the model worded carries {"generated": true} in place of its text. The names are each agent’s AgentSpec.name, else the call’s agent_id, on both frameworks.
Coval shows conversation, turn, llm and llm_tool_call with their own colours and runs its built-in metrics on them. Langfuse and Logfire type spans from gen_ai.operation.name, not from the span name.
Vendor names
The span contract above is neutral. A vendor’s own names are added inside that vendor’s sink, on a copy of each span just before export. No sink changes what another sink gets.Coval
Text keys move to the Coval name, so a large prompt is not sent twice. Other keys stay. Ametrics.ttfb of zero or less is dropped, because Coval would average it as an instant answer.
Coval names by span
Coval names by span
prompt.message_count is how many there were, and prompt.messages_traced is how many were kept.
There is no vad span and no transport span. The turn carries vad.end_of_turn_delay, and the call span carries the origin and the correlation method, so they hold what those spans would say. The call span is still held until the ID is known.
Logfire
Logfire adds logfire.msg, the row label: turn: <first 80 characters of what the caller said>, chat <model>, tool <name>, stt <model>, tts <model> and call <session id>. On a model request it also writes the messages under the GenAI names Logfire’s LLM panel reads: gen_ai.system_instructions, gen_ai.input.messages (tool calls and tool replies as parts) and gen_ai.output.messages. input.value and output.value stay. With filter=MetadataOnly() the filter runs first, so only the labels go out.
Where to go next
Telemetry
Send a call’s spans to Langfuse, Logfire, Coval or Datadog.
Dev records
What one call did, for the local dev page.