Add opt-in OpenTelemetry tracing for workflow execution

[hertznsk]

Summary

Add opt-in OpenTelemetry tracing to Conductor so workflow authors can inspect orchestration behavior, state transitions, step execution, failures, latency, token usage, cost, and tool calls without reconstructing a run from CLI output.

With runtime.telemetry.enabled: true and a standard OTEL_EXPORTER_OTLP_ENDPOINT, an OTLP-compatible backend such as Jaeger, Tempo, or Grafana should receive an orchestration span tree shaped like:

workflow
├── agent / validator / script / set / wait / gate
├── parallel / for_each / subworkflow
│   └── member or item spans
└── tool calls

For the Pydantic AI-backed claude and openai providers, also enable native Pydantic AI instrumentation. Those native LLM traces should use the same TracerProvider and be correlated with the orchestration trace through gen_ai.conversation.id = run_id.

Motivation

The current CLI and event-log surfaces are useful but require manual reconstruction and do not expose every timing and state transition in a form suitable for distributed-tracing tools. OpenTelemetry would provide:

• detailed debugging of workflow control flow and failure paths;
• performance analysis across agents, groups, subworkflows, and tools;
• identification of latency and cost bottlenecks;
• data for subsequent prompt and workflow optimization;
• a standards-based integration surface rather than a backend-specific telemetry implementation.

Proposed configuration

Telemetry is disabled by default and enabled per workflow:

workflow:
  runtime:
    telemetry:
      enabled: true
      service_name: conductor   # optional

Exporter and SDK behavior should use standard OpenTelemetry environment variables, including:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=conductor

The SDK/exporters should ship as an optional telemetry extra. A workflow with telemetry enabled but the extra unavailable should warn with the correct install command and continue without spans.

Design

Event-driven orchestration tracing

Build orchestration spans in a dedicated TelemetrySubscriber over the existing WorkflowEventEmitter stream rather than adding tracing calls throughout the engine hot path.

The subscriber should:

• create detached spans with explicit parent contexts because event callbacks do not own the async task context between paired events;
• preserve event timestamps, converting Unix seconds to the nanoseconds expected by the OTel SDK;
• track open spans by stable identities for workflows, nested contexts, agents, parallel members, for-each item indexes, validators, and tool calls;
• close and remove every span on success, failure, cancellation, workflow failure, or subscriber shutdown;
• handle duplicate workflow_started events during resume without creating duplicate roots;
• receive the directly synthesized root event used by the resume --web path, where the engine root emission is suppressed;
• close gate presentation and resolution as separate short spans rather than holding a span open while waiting for a human;
• flush on shutdown with a bounded force_flush wait and explicit OTLP exporter timeout.

Use OpenTelemetry GenAI semantic conventions where applicable:

• gen_ai.operation.name: invoke_workflow, invoke_agent, execute_tool;
• gen_ai.conversation.id;
• gen_ai.agent.name, gen_ai.tool.name;
• gen_ai.provider.name, gen_ai.request.model;
• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens;
• standard error.type on failures.

Conductor-specific data should remain under conductor.*, for example cost, step type, group name, item key, iteration, resumed, and superseded state.

Pydantic AI native instrumentation

For claude and openai, enable Pydantic AI instrumentation only while Conductor telemetry is active. Pass the Conductor run_id explicitly as conversation_id to Pydantic AI execution calls and provide the active TracerProvider explicitly.

The v1 contract is intentionally two correlated trace trees, not one parented tree:

1. the Conductor orchestration trace;
2. native Pydantic AI traces for model and tool internals.

They share gen_ai.conversation.id = run_id and the same provider, but cross-layer context parenting is out of scope for v1. Documentation must state this clearly.

Validator agents should receive the same native instrumentation as primary agents because their latency and cost are part of the run.

Privacy

Prompt and response content must not be captured by default. Content capture is controlled only by the standard OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT environment variable, with no Conductor-specific YAML switch.

Pydantic AI instrumentation must be configured with an explicit include_content policy. In particular, do not use a bare global instrumentation switch whose upstream default captures content.

Scope

Included

• OTLP export over gRPC or HTTP/protobuf;
• optional telemetry dependency extra;
• workflow-level runtime.telemetry configuration;
• orchestration spans for workflow, agent, group, item, step, validator, gate, and tool lifecycle events;
• native Pydantic AI instrumentation for claude and openai;
• run/resume parity, including dashboard resume event seeding;
• validation warning when the optional SDK is unavailable;
• an example workflow and documentation for Jaeger/Tempo/Grafana setup, environment variables, privacy, and experimental GenAI semantic conventions.

Out of scope for v1

• a new telemetry CLI command;
• ~/.conductor/config.toml settings;
• dashboard integration;
• cross-layer parenting that merges Conductor and Pydantic AI spans into one trace;
• native provider telemetry for Copilot, Claude Agent SDK, Hermes, or ACA beyond orchestration-level attributes already available from Conductor events;
• prompt-content capture unless explicitly enabled through the standard OTel environment variable;
• vendor-specific exporters other than OTLP.

Acceptance criteria

• Telemetry remains a silent no-op by default: no spans, warnings, or behavior changes when runtime.telemetry is absent or disabled.
• runtime.telemetry.enabled: true plus an OTLP endpoint exports a workflow-to-agent/group/item/tool orchestration tree.
• All success, failure, cancellation, fail-fast, and resume paths leave no open spans.
• Resume with and without --web produces the correct root span and preserves the run correlation identifier.
• For-each items remain distinct even when authored item keys collide; item index is available in every relevant per-item event payload.
• Repeated same-name tool calls produce distinct spans and retain the correct agent/item parent.
• claude and openai emit native Pydantic AI traces through the same TracerProvider, with gen_ai.conversation.id equal to the Conductor run_id.
• Documentation describes the native traces as a second correlated tree, not a single end-to-end trace.
• Prompt and response content is absent from spans by default and appears only when explicitly enabled via the standard OTel environment variable.
• Missing telemetry dependencies, OTEL_SDK_DISABLED=true, exporter initialization failure, and an unreachable endpoint never crash or block workflow completion beyond the documented flush/export timeout.
• Documentation includes an OTLP quickstart, privacy guidance, semantic-convention stability notice, configuration reference, and a validated example workflow.
• Unit and integration tests use an in-memory span exporter and cover normal execution, agent failure, resume, fail-fast cancellation, for-each key collision, tool spans, disabled SDK, missing extra, content policy, and exporter degradation.