Skip to main content
This page is the full detail behind Telemetry. Start there to send a call’s spans somewhere. Come here to call the helpers by hand, write a sink, or read a span’s attributes.

How it works

  1. The LiveKit or Pipecat adapter calls the start_* and end_* helpers on session events.
  2. Telemetry owns one TracerProvider for the call. It stamps the call’s identity on every span.
  3. Each sink adds a batch processor and an OTLP/HTTP exporter to that provider.
  4. A sink with filter=MetadataOnly() sends no text. It keeps only an allow-list of attributes.
  5. close() ends the call span and flushes every sink.

Telemetry

Telemetry arguments

Sink
required
Where the spans go. Pass at least one.
str
required
The agent’s name. It is the trace name and the service name.
str
required
The call’s ID, like the room or session ID. It is stamped on every span as session.id. An empty agent or call_id raises ValueError.
Mapping[str, AttributeValue] | None
default:"None"
Extra resource attributes, like deployment.environment.name.
Sequence[SpanProcessor]
default:"()"
Extra span processors that run before the sinks, for example one that adds your own attributes.
Set[str]
default:"{\"openrouter\", \"slng\"}"
Providers that pass a request on to another provider named in the model id, like OpenRouter. A speech span for one of them records the provider behind it as unmute.upstream.provider. For slng/fish/tts:s2.1-pro the value is fish. Add your own gateway here. The default is GATEWAYS.

Call the helpers by hand

The adapters call these helpers for you. Call them yourself only on a framework the library has no adapter for.
agent.py

Helpers

close waits up to 10 seconds by default (CLOSE_TIMEOUT_SECONDS) so a call that no simulation placed has time to be filed with Coval. It returns whether the flush finished. Rules for the end helpers:
  1. Latencies are in seconds. A missing, negative or infinite latency is left out, never sent as 0.
  2. error="<type>" sets error.type and the span status to ERROR. On Pipecat, the llm span gets the error type of an ErrorFrame from the LLM service.
  3. An attribute with no value is not set.
  4. gen_ai.provider.name is lowercase, so one provider has one name on both frameworks.
  5. A text attribute keeps up to 256 KiB, so a long prompt stays valid JSON.
describe_call(origin=..., framework=...) sets unmute.call.origin (phone, browser or websocket) and unmute.call.framework (livekit or pipecat) on the call span. Values given before start_call wait and are set when it starts. After the call span ended, it does nothing. The adapters stamp the framework themselves. correlate_coval sets the origin.

On a plain agent

On LiveKit, UnmuteAgent traces the session for you. trace_session is for a plain Agent:
A native LLM plugin on a plain Agent gets an llm span through traced_llm, which also fills {{placeholders}} from the trace’s state. tts spans need a tts_node on your agent:
ModelCallLLM(model, trace=trace) also gives an llm span when a ModelCall answers. On Pipecat, leave Pipecat’s own enable_tracing off, or the call is traced twice. Set PipelineParams(enable_metrics=True, enable_usage_metrics=True), or the spans have no time to first byte and no tokens. TelemetryObserver logs one warning when these are not on.

TelemetryObserver

Telemetry
required
The call’s telemetry, with its call span started.
str
required
The model name the llm spans carry. After a handoff gives the next agent its own model call, its spans name that one.
str
required
Who serves the model, like openai.
UnmuteFlows | None
default:"None"
The call’s tasks, which tell each request’s prompt site.
With a ModelCallLLMService in flows, the observer also sets the service’s trace_headers. Each model request then carries a traceparent, so the router’s spans join the call’s trace. On Pipecat they are children of the turn span, not of the request’s llm span. The observer gets frames through a queue, so the llm span may open after the request is sent.

Sinks

Each of the first three has from_env() and from_settings(settings). from_env raises TelemetryConfigError and names every missing variable. A missing Coval key only logs a warning. LangfuseSettings, LogfireSettings and CovalSettings read the same variables. Pass one to from_settings to load them at startup with check_settings, or to set values in code. Both methods take sink options too, like Langfuse.from_env(filter=MetadataOnly()). Settings lists every variable.

Sink arguments

str
required
Langfuse. The project’s public key.
str
required
Langfuse. The project’s secret key.
str
required
Langfuse. Like https://cloud.langfuse.com.
str
required
Logfire. A write token.
str | None
required
Coval. The Coval API key. None exports nothing.
str | None
default:"None"
Coval. The simulation ID for a local run. An adapter’s correlate_coval reads it.
bool
default:"True"
Coval. File a call no simulation placed as a Coval conversation at the end. False files nothing.
str | None
default:"None"
Otlp. The full traces URL, like http://collector:4318/v1/traces. With none, the OpenTelemetry SDK reads its own OTEL_EXPORTER_OTLP_* variables.
dict[str, str]
default:"{}"
Otlp. Extra HTTP headers, like an API key.
MetadataOnly | None
default:"None"
Every sink. Lets only metadata through.

Datadog

Send the spans with an Otlp sink to an OpenTelemetry Collector. The Collector exports them to Datadog. The library needs no Datadog package.
A Collector config that receives OTLP over HTTP and exports to Datadog:
collector.yaml
Set site to your Datadog site, like datadoghq.eu. Keep the API key in the Collector’s environment, not in the agent. Add filter=MetadataOnly() to the Otlp sink to send no text.

Coval

Coval needs to know which Coval run the call belongs to. Spans wait in memory, up to 1024, until coval.activate(simulation_id=...) or coval.activate(conversation_id=...). Pass method="dispatch_metadata" or similar to record how the ID was found. It becomes coval.correlation.method on the call span. activate raises ValueError unless you pass exactly one of the two IDs. It raises RuntimeError when no Telemetry was built with the sink, or the call already has a Coval ID. It sends the held spans on the calling thread, so call it with asyncio.to_thread from async code. An adapter finds the ID for you. Call correlate_coval once, with the sink the call’s Telemetry was built with:
On Pipecat, runner_args is Pipecat’s RunnerArguments. It also tells telemetry how the caller arrived, with describe_call(origin=...). The ID is the coval.simulation_id key, or any key that ends in x-coval-simulation-id, in any case. coval_simulation_id(mapping) does the lookup. On LiveKit the first ID wins, and the sink activates once on a worker thread. On Pipecat it activates at once, on the calling thread. A call that no simulation placed is filed when it ends. A real call or a local run then shows up in Coval’s trace search.
  1. Telemetry.close() flushes, then the provider shuts the Coval exporter down.
  2. The exporter reads the transcript, the session ID, the agent name and the origin from the held call span. It merges consecutive lines of one speaker.
  3. It posts them to https://api.coval.dev/v1/conversations:submit and waits at most 5 seconds.
  4. With the conversation ID it sends every held span with the X-Conversation-Id header. The call span gets coval.conversation_id and coval.correlation.method=conversation_submit.
An adapter logs the two events with helpers you can reuse in your own adapter. log_no_simulation(log, call="call") says no simulation placed the call, so its trace is filed at the end. log_activated(log, simulation_id, method, call=None) says the sink is now bound to a simulation. A failed submit logs a warning and never raises. The held spans are then dropped. A call is not filed, and the log says why, when:
  • it already has a simulation or conversation ID,
  • there is no Coval API key,
  • the sink is Coval(file_calls=False),
  • the sink has a filter, because metadata only never sends the transcript,
  • the call span has no transcript.

Metadata only

The filter does the following:
  1. It keeps an attribute only if the key is on the list and the value is a number, a bool or a short safe string.
  2. It renames any span not on the list to operation.
  3. It drops events, link attributes and the status description.
  4. It keeps only allowed resource keys.

MetadataOnly arguments

frozenset[str]
default:"DEFAULT_ATTRIBUTES"
Span attribute keys to keep. Pass DEFAULT_ATTRIBUTES | {"my.key"} to allow more.
frozenset[str]
default:"DEFAULT_SPAN_NAMES"
Span names to keep.
frozenset[str]
default:"DEFAULT_RESOURCE"
Resource attribute keys to keep.
MetadataOnly is a frozen Pydantic dataclass. A value of the wrong type raises a ValidationError. filter.apply(span) filters one span. filter.apply_all(spans) filters a batch. The sink uses apply_all on each export. It builds the filtered resource once for each resource the spans share, so a batch of a call’s spans costs less than one apply per span.

Write a sink

Sink is the plug point for a new provider. Subclass it and return an exporter:
The base class wraps the exporter in a batch processor, and in filter when one is set. Build sinks once per call, with the call’s Telemetry. Sink is a Pydantic dataclass. A stdlib @dataclass(kw_only=True) subclass, as above, works. So does a pydantic.dataclasses.dataclass(kw_only=True) subclass. With the Pydantic one, a field of the wrong type, like api_key=3, raises a ValidationError when you build the sink. The built-in sinks check their fields the same way.

Span contract

Both adapters emit these spans through the helpers. Do not create them by hand.
  1. conversation is the call. It starts a new trace. When it closes, its input is the transcript, one entry per line. Its output is how the call ended: {"end_reason", "state", "confirmed"}.
  2. turn is one caller message and everything the agent does until the next one. Its output is every line the agent said, joined by a space. Only a turn where the agent speaks first has no caller input.
  3. stt is one transcript. tts is one piece of speech.
  4. llm is one model request: the prompt as filled, the tools offered, and the text and tool calls the model chose. llm_tool_call is one tool run.
  5. An event span is one decision, like a task starting, a step skipped or a handoff. It has no length. Langfuse does not show OpenTelemetry span events, so each one is a span.
  6. task_step sits under the call, not around turns. One turn can finish a task and start the next.
Every span also gets session.id, gen_ai.conversation.id (both the call ID) and agent.name. The resource has service.name set to the agent name.
On LiveKit, STT is read from the caller’s message, not from stt_node. Overriding stt_node would stop LiveKit reusing the STT stream each time a step starts. A subclass that overrides tts_node loses the tts spans unless it calls super().tts_node(...). LiveKit’s own spans stay off, and the adapter does not call set_tracer_provider.

What a tool span shows

A tool that takes no parameters shows {} as its input. A tool span’s output is what the tool gave back, in three cases that need care:
  1. A task call’s span lasts the whole task. Its output is the task result the main agent reads, not Pipecat’s first “running” result.
  2. A step’s closing tool shows what it really returned, like the customer record. The model reads only how the step ended, and the task_step span shows that.
  3. A handoff shows {"status": "handed off"} on both frameworks. A step that hands off ends its task_step span with the outcome handed_off.

Model requests that make no reply

Three kinds of model request make no reply. All are still llm spans in the turn they belong to.
  1. A step’s finish check. Its tools are only finish and not_yet, and a call is required. A not_yet is an answer, never run, so it has no tool span. A finish it chose runs as a tool, under the call id the check gave it.
  2. The decision on a held save, made when the call ends. It sits in the last turn. LiveKit settles before it ends the turn, and a Pipecat host calls flows.settle() before the EndFrame. Its calls are answers too, never run.
  3. A summary, for a handoff or a step with history: summary. Its site is summary, it offers no tools, and its output is the summary text.
See Tasks and Handoffs for when each runs.

When a turn starts and ends

A turn belongs to the caller until the agent says a line, starts speech or calls a tool. A model request alone does not count. A request cut off because the caller went on speaking leaves the rest of their words in the same turn. More caller input before that joins the open turn, like a caller who pauses mid-sentence. Caller input after that starts the next turn.

Following the turns in your own adapter

Both adapters use TurnTracker. It numbers the turns and keeps the open turn span. You decide when a turn starts and what its text and latencies are.
method
start(). Opens the next turn under the call and returns its span. End the earlier turn first.
method
open(answering=True). Returns the open turn, or opens one when none is. It marks the turn answered unless you pass answering=False.
bool
True while a turn is open and nothing the agent did has answered in it.
method
end(**fields). Ends the open turn through Telemetry.end_turn. It does nothing when no turn is open.
TypedDict
Names the fields of end: user_text, agent_lines, interrupted, e2e, stt_final, eou, llm_ttft, tts_ttfb and error. They are the same as the arguments of end_turn.

Events

The events are the task stages (task.started, task.completed, task.refused_busy, group.step_skipped, task.handed_off and the rest of TaskStage), handoff with {"from", "to"}, and the idle nudges (idle_nudges.armed, idle_nudges.first.fired and the rest). A nudge the model worded carries {"generated": true} in place of its text. The names are each agent’s AgentSpec.name, else the call’s agent_id, on both frameworks. Coval shows conversation, turn, llm and llm_tool_call with their own colours and runs its built-in metrics on them. Langfuse and Logfire type spans from gen_ai.operation.name, not from the span name.

Vendor names

The span contract above is neutral. A vendor’s own names are added inside that vendor’s sink, on a copy of each span just before export. No sink changes what another sink gets.

Coval

Text keys move to the Coval name, so a large prompt is not sent twice. Other keys stay. A metrics.ttfb of zero or less is dropped, because Coval would average it as an instant answer.
Coval rejects an oversized batch with a 413, so text is cut. Most text keeps 1,024 characters. A prompt keeps 8,192. The system messages stay and the newest other messages fill the rest. prompt.message_count is how many there were, and prompt.messages_traced is how many were kept. There is no vad span and no transport span. The turn carries vad.end_of_turn_delay, and the call span carries the origin and the correlation method, so they hold what those spans would say. The call span is still held until the ID is known.

Logfire

Logfire adds logfire.msg, the row label: turn: <first 80 characters of what the caller said>, chat <model>, tool <name>, stt <model>, tts <model> and call <session id>. On a model request it also writes the messages under the GenAI names Logfire’s LLM panel reads: gen_ai.system_instructions, gen_ai.input.messages (tool calls and tool replies as parts) and gen_ai.output.messages. input.value and output.value stay. With filter=MetadataOnly() the filter runs first, so only the labels go out.

Where to go next

Telemetry

Send a call’s spans to Langfuse, Logfire, Coval or Datadog.

Dev records

What one call did, for the local dev page.