Skip to content

test: backend coverage parity audit post-refactor #1054

Open

Description

@ajbozarth

Background

Refresh of retired sub-issue #477. After the plugin refactor, verify all backends emit spans uniformly and parity holds across openai, ollama, huggingface, watsonx, litellm (and vllm if applicable).

Scope

  • Add integration tests (gated on backend-specific markers, per test/MARKERS_GUIDE.md) asserting each backend produces backend-generation spans with a matching set of gen_ai.* attributes.
  • Document any backend-specific attribute gaps.
  • Cross-check the Anthropic-specific cache/reasoning token attributes emitted by PR feat(telemetry): close five OTel GenAI semantic convention emission gaps (#1035) #1036 (gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens, gen_ai.usage.reasoning.output_tokens) against the current OTel GenAI semconv at audit time; rename to match the spec if equivalents have been standardised, otherwise flag with a code comment as Anthropic-specific extensions pending standardisation. (Per @planetf1's comment on Epic: Enhanced distributed tracing #444.)

Phase & dependencies

Phase 3 (polish). Depends on #1045 and #1046. Can start any time after Phase 1 lands; best done after most of Phase 2 has merged so the audit is against steady-state tracing.

Acceptance criteria

  • All backends produce backend-generation spans with a shared minimum attribute set.
  • Known gaps documented.
  • Anthropic-specific cache/reasoning token attribute names cross-checked against OTel GenAI semconv; renamed or flagged as documented above.

Parent epic: #444

Activity

  1. added
    area/backendsProvider-specific work: Ollama, HF, LiteLLM, OpenAI, Bedrock, vLLM
    area/telemetryOTel spans, metrics, tracing, semconv
    on May 20, 2026
  2. added
    p2Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.
    on Jun 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/backendsProvider-specific work: Ollama, HF, LiteLLM, OpenAI, Bedrock, vLLMarea/telemetryOTel spans, metrics, tracing, semconvp2Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.testing

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions