[exploration] Voice schema option: span points

[kev (kevchoi)]

AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-10-13.

This is an exploration draft: option 4 of the voice-trace schema options kept for review. It is not for merge. It is stacked on #2612 (call timeline), so the diff shows only what span points change.

Problem

Call timeline (#2612) gives one precise log for latency, overlap and missing evidence. But the log sits in voice.timeline rows of its own, away from the turn, LLM and tool spans a customer browses. Two cases also stay implicit:

• speech that nothing answered;
• whether an overlap was a barge-in or the caller talking over a reply that kept going.

Proposed contract

Integrations log #2612's events as points on the ordinary span that observed each one, in metadata["voice.points"], a map keyed by the per-call seq:

Owner span	Points
user_turn	caller speech, transcripts, end_of_turn and the caller message
assistant_turn	agent playout, played text, truncate and the agent message
llm_response	response_start
tool span	tool_call / tool_result
call span	voice.points_version: 1 and call_end

• Owner rule: a point goes on the span of who observed it, never on whatever operation was running. Caller speech never lands on a tool or TTS span, and playout never on llm_response (generation) or tts (synthesis).
• Owner-supplied fields: the owner provides what Voice schema option: call timeline #2612 carried as fields: span_id for generation and tools, and the tool name.
• Additions:
  • speech_end.interrupted_by names the caller utterance that cut the agent off.
  • A user_turn with no end_of_turn marks speech nothing answered.
• One or many points per span: many. A single point per span would need about 30 rows for the 13 s call, or drop facts.
• A map, not an array: Braintrust merges objects by key but replaces arrays. One key per write is idempotent under retry and independent of arrival order.
• Late writes: go to the original owner, even after it ended. Points are never rewritten.
• Limit: 256 per span, with overflow reported on the call span.
• Completeness: seq gaps across spans and call_end.

Main tradeoff

The evidence and answers are #2612's, and the raw trace shows speech on the turn spans again. What changes:

• Gains: interrupted_by separates barge-in from talk-over without a heuristic, and an unanswered turn is visible.
• Dependence on turn spans: capture depends on the user_turn / assistant_turn spans the LiveKit and Pipecat reference integrations already emit. Points fall back to the call span when an owner is missing.
• Write cost: about 30 one-key merges for the example call, instead of 2 chunk rows.
• Index cost: each seq key is its own JSON path, which needs a Brainstore owner's check.

LiveKit 1.9.2 cascaded tool replies reuse the speech handle (agent_activity.ts:4391-4421), so llm-1 and llm-2 both start speech_1 here. #2612's cascaded example gives the reply its own handle; #2612 is unchanged.

Implemented vs sketched

• Implemented:

  • the zod contract, including the owner table (js/src/instrumentation/livekit/voice-schema/span-points.ts, not exported);
  • cascaded and realtime examples (docs/voice-schema/span-points/examples/: 14 and 11 rows, 30 and 28 points). They have normal LLM and tool inputs and outputs, and a recording attachment with channel sources. The cascaded example adds a barge-in and a separate talk-over under a non-interruptible reply;
  • a test that parses the examples and checks seq across spans, owners, one owner per utterance, start/end pairing, interrupted_by, and barge-in versus talk-over.

• Sketched in docs/voice-schema/span-points/README.md:

  • LiveKit owner lookup and writes;
  • Pipecat owners (the reference integration's turn spans);
  • edge cases and open questions.

  No instrumentation emits points yet.

• Lingua mapping: implemented in the companion PR, [exploration] Voice schema option: span points (Lingua mapping) lingua#495.

Checks

• pnpm exec vitest run src/instrumentation/livekit/voice-schema (in js/): 21 passed, 11 of them new.
• pnpm exec tsc --noEmit -p .: passed.
• pnpm exec eslint on span-points.ts: passed. The test file is outside the lint config.
• prettier --check on the new docs, examples and sources: passed.
• An independent agent read the README and spec against the examples, the zod, the Lingua code and the LiveKit 1.9.2 sources, and its findings were fixed.
• Skipped: SDK e2e (no instrumentation changed) and Pipecat/Python code.

Companion PR: braintrustdata/lingua#495

🤖 Generated with Claude Code