Skip to content

fix(observability): match the collector's real Tempo exporter label - #1765

Merged
mashkovd merged 1 commit into
mainfrom
fix/tempo-eval-collector-exporter-label
Oct 10, 2026
Merged

mashkovd merged 1 commit into
mainfrom
fix/tempo-eval-collector-exporter-label

Conversation

@mashkovd

Copy link
Copy Markdown
Member

Found during the synthetic span proof after #1764 (2026-10-10).

The live otel-collector (0.160) exposes the Tempo exporter's telemetry as exporter="otlp_grpc/eval-tempo", the resolved type, not the config ID otlp/eval-tempo:

otelcol_exporter_sent_spans{exporter="debug"}                 72
otelcol_exporter_sent_spans{exporter="otlp_grpc/eval-tempo"}  72

TempoEvalCollectorExportFailures and TempoEvalCollectorDroppedSpans selected exporter="otlp/eval-tempo", so they had no series to evaluate and could never fire. The unit tests fed the same wrong label, so they passed anyway.

Changes:

  • Both rules now select exporter=~"otlp(_grpc)?/eval-tempo", so a collector upgrade that changes the alias cannot silently blind them.
  • The tests feed the real label otlp_grpc/eval-tempo. The "other exporter stays quiet" cases are unchanged.
  • The runbook's verification queries use the real label, with a note.

Verified both ways with scripts/check-vm-rules.sh (promtool): green with this change, and export failures to Tempo fire goes red when the old exact selector is restored against the real label.

Refs #1280

🤖 Generated with Claude Code

https://claude.ai/code/session_013zqBqP9sSbVTd1kXRLA6AU

The live collector (0.160) labels its telemetry with the resolved
exporter type, otlp_grpc/eval-tempo, not the config ID otlp/eval-tempo.
Both collector-side Tempo alerts selected the config ID, so they had no
series to evaluate and could never fire. The tests used the same wrong
label, so they passed anyway.

Claude-Session: https://claude.ai/code/session_013zqBqP9sSbVTd1kXRLA6AU
Copilot AI balanced review requested due to automatic review settings October 10, 2026 11:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@claude

claude Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude finished @mashkovd's task in 1m 9s —— View job


Review: tempo eval collector exporter label fix

  • Gather context (diff 403f0771...9a04c699, runbook, VMRule, and test file)
  • Verify the regex selector matches the real collector label without widening to other exporters
  • Check runbook/test consistency with the new label
  • Post findings and submit verdict

Analysis

The fix is sound. otlp(_grpc)?/eval-tempo is a correctly anchored regex matcher (VM/Prometheus =~ is full-string match), so it catches both the config ID (otlp/eval-tempo) and the collector's resolved-type label (otlp_grpc/eval-tempo) without matching unrelated exporters — confirmed the "different exporter stays quiet" test still uses otlp/backend, which the new pattern doesn't touch.

Tests in tempo-eval-alerts_test.yaml now feed the real otlp_grpc/eval-tempo label for both TempoEvalCollectorExportFailures and TempoEvalCollectorDroppedSpans, matching the annotation text update. Runbook verification queries and the scoring table were updated to the same real label.

One pre-existing line not touched by this diff (tempo-eval-alerts.yaml:7, file-level header comment: "...the otlp/eval-tempo exporter") is now slightly stale since the selector matches two labels — not part of this diff's hunks so no inline comment was possible, purely optional follow-up.

No P1/P2 findings. 0 P3 (the header-comment note above is informational, not a blocking nit).

@github-actions

Copy link
Copy Markdown
Contributor

Antigravity (agy) review — informational

Model: gemini-3.8-flash-medium · commit 9a04c69.

No significant issues found.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No P1/P2 findings (0 P3). Regex selector correctly matches both the config-ID and collector-resolved exporter labels without widening to other exporters; tests and runbook updated to the real label. Good to merge.

@mashkovd
mashkovd merged commit e1c4d23 into main Oct 10, 2026
24 checks passed
@mashkovd
mashkovd deleted the fix/tempo-eval-collector-exporter-label branch October 10, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants