Smoke OTEL - 32577789236

[github-actions]

Executive Summary

Overall: ❌ FAIL

The current run (32577789236) emits OTEL spans locally and the write-side configuration for all three remote backends is present, but only the local mirror could be confirmed end-to-end. Sentry could not confirm this run's spans (though recent gh-aw telemetry is visible — inconclusive). Grafana's MCP surface exposes only datasource discovery tools, with no trace-search capability to confirm read access (fail). Datadog's OTLP export failed with a 404 and its MCP backend returned 503/unavailable (fail).

Results Matrix

Backend	Write Config Present	Write Export Succeeded	Read Config Present	Read Query Succeeded	Overall
Local OTLP	✅	✅	—	—	✅
Sentry	✅	✅	✅	🔶	🔶
Grafana	✅	✅	✅	❌	❌
Datadog	✅	❌	❌	❌	❌

Evidence

Step 1 — Local OTLP (pass)

• GH_AW_OTLP_ENDPOINTS was set, listing three configured backend hosts: o205451.ingest.us.sentry.io, otlp-gateway-prod-eu-west-2.grafana.net, otlp-intake.datadoghq.eu.
• /tmp/gh-aw/otel.jsonl exists (1 line) and contains a gh-aw.agent.setup span whose resource attributes include gh-aw.run.id = github.run_id = "32577789236" — the current run is confirmed present in the local mirror.
• /tmp/gh-aw/otlp-export-errors.count = 1; /tmp/gh-aw/otlp-export-errors.jsonl contains exactly one entry: {"host":"otlp-intake.datadoghq.eu","status":404,"reason":"Not Found"}. This error is attributable only to Datadog (see below); it does not affect the local-mirror pass verdict.

Step 2 — Sentry (inconclusive)

• sentry whoami succeeded (authenticated as Mara Kiefer via the MCP service account) — MCP connection and query access work.
• sentry find_organizations found org github; sentry find_projects --organizationSlug github --query gh-aw found project gh-aw (project id 4511347087179777).
• Query for current-run spans (dataset=spans, query="span.description:*32577789236*", statsPeriod=24h) returned No results found.
• Fallback query for recent gh-aw spans (same dataset, no filter, statsPeriod=24h) returned multiple recent spans, e.g. POST /mcp/tavily / gateway.request, trace c255beb23f4a06ca954e4bac0e7a5bd1, timestamp 2026-08-22T14:11:30Z (https://github.sentry.io/explore/traces/trace/c255beb23f4a06ca954e4bac0e7a5bd1). This confirms the Sentry read path and recent gh-aw telemetry ingestion are healthy, but the current run's specific spans are not yet visible (ingestion delay, not a broken query path).
• errors and logs datasets for the same project/window returned No results found (explicit empty-dataset check, not treated as failure).
• No OTLP export errors attributable to Sentry were recorded in otlp-export-errors.jsonl.

Step 3 — Grafana (fail)

• grafana list_datasources succeeded and returned a Tempo tracing datasource: grafanacloud-traces (uid grafanacloud-traces, type tempo), confirming MCP connectivity and read-config presence.
• grafana get_datasource --uid grafanacloud-traces returned full datasource details (Tempo backend at tempo-prod-10-prod-eu-west-2.grafana.net), confirming the tracing surface is reachable and configured.
• However, the Grafana MCP tool surface exposed in this workflow only provides get_datasource and list_datasources — there is no trace-search or query-execution tool (e.g. no TraceQL/Tempo search command) to actually query spans for 32577789236 or recent gh-aw traces. grafana --help confirms only 2 commands are available.
• No OTLP export errors attributable to Grafana were recorded in otlp-export-errors.jsonl, so write-side export is presumed to have succeeded, but the read query could not be executed at all.

Step 4 — Datadog (fail)

• The configured OTLP backend host list includes otlp-intake.datadoghq.eu, confirming write-config presence.
• /tmp/gh-aw/otlp-export-errors.jsonl records one entry attributable to Datadog: {"host":"otlp-intake.datadoghq.eu","status":404,"reason":"Not Found"} — the write/export path failed for this run.
• datadog list_tools and datadog --help both failed: MCP initialize returned status=503, and the tool call returned {"error":"backend_unavailable","message":"Backend MCP server is not ready; retry initialization"}. A retry after a short delay produced the identical 503 result, and --help reported Commands (0): (tool list unavailable) — the Datadog MCP backend is not operational, so read-config/read-query cannot be confirmed.

Failure Analysis

Grafana — Read Query Succeeded: ❌

• Failing step: Step 3, attempting to search Tempo for spans matching 32577789236 or recent service.name=gh-aw traces.
• Evidence observed: grafana --help lists only get_datasource and list_datasources; no trace-search/query tool exists in this MCP CLI surface. get_datasource confirms the Tempo datasource (grafanacloud-traces) is configured and reachable, but that call cannot return span/trace data.
• Most likely root cause: The Grafana MCP server exposed to this workflow was configured with a restricted/incomplete tool set that omits trace-query capability (e.g. a "query" or "search_traces" tool was not registered), rather than an auth or connectivity problem — connectivity and datasource discovery both succeeded.
• Layer: Read path / MCP tool configuration (not auth, not the backend itself — Tempo appears reachable).
• Next step: Add/enable a trace-query tool (e.g. Tempo TraceQL search or a generic "query datasource" tool) to the Grafana MCP server configuration used by this workflow, then re-run this smoke test to confirm current-run span visibility in Tempo.

Datadog — Write Export Succeeded: ❌

• Failing step: Step 1/4, OTLP span export to otlp-intake.datadoghq.eu.
• Evidence observed: otlp-export-errors.jsonl records {"host":"otlp-intake.datadoghq.eu","status":404,"reason":"Not Found"} for this run.
• Most likely root cause: A 404 on an OTLP intake endpoint typically indicates an incorrect endpoint path (e.g. missing /api/v2/otlp or similar suffix) or a datacenter/region mismatch (.eu vs .com intake) rather than an auth failure (which would be 401/403).
• Layer: Write path / configuration (endpoint URL), not auth or the Datadog backend itself.
• Next step: Verify the exact OTLP intake path expected by Datadog for the datadoghq.eu site (check GH_AW_OTEL_DATADOG_ENDPOINT against Datadog's documented OTLP HTTP intake path) and correct the endpoint construction in the OTLP exporter configuration; re-run and confirm the export error disappears from otlp-export-errors.jsonl.

Datadog — Read Config Present: ❌ / Read Query Succeeded: ❌

• Failing step: Step 4, datadog list_tools / any Datadog MCP tool call.
• Evidence observed: MCP initialize returned status=503 with no sessionId; the subsequent tool call returned {"error":"backend_unavailable","message":"Backend MCP server is not ready; retry initialization"}. A retry after ~5s produced the same result, and datadog --help reported 0 available commands.
• Most likely root cause: The Datadog MCP server backing this CLI bridge is not running or not finished initializing in this environment — a service/infrastructure availability issue on the MCP gateway side, separate from the OTLP export 404.
• Layer: Read path / backend infrastructure (MCP server availability), not query syntax or auth (no auth challenge was reached before the 503).
• Next step: Confirm the Datadog MCP server process/container is deployed and healthy for this workflow's MCP gateway (awmg-mcpg), then retry datadog list_tools to confirm tool availability before re-attempting the read-query check.

Sentry — Read Query Succeeded: 🔶 (inconclusive, not a failure)

• Failing step: N/A — query access and dataset access both work; only the current-run-specific filter returned no rows.
• Evidence observed: span.description:*32577789236* returned no results in the spans dataset for the gh-aw project over the last 24h, while an unfiltered query against the same dataset/project/window returned multiple recent spans (trace c255beb23f4a06ca954e4bac0e7a5bd1).
• Most likely root cause: Export/ingestion latency between the run's OTLP export and Sentry's indexing, or the current run's spans use different span.description/attribute values than assumed and require an alternate filter (e.g. by trace ID or github.run_id field) rather than a free-text span.description match.
• Layer: Propagation/timing (most likely) or query-filter mismatch (secondary possibility).
• Next step: Re-run this same Sentry query a few minutes after the workflow completes, and additionally try filtering on the exact resource attribute key (e.g. github.run_id:32577789236) rather than span.description, to rule out a filter-field mismatch before concluding this is purely an ingestion-delay issue.

Run

https://github.com/github/gh-aw/actions/runs/32577789236

Generated by 🧪 Smoke OTEL · auto · 36.1 AIC · ⌖ 2.66 AIC · ⊞ 11.5K · ◷
_{Comment /smoke-otel-backends to run again
Add label smoke to run again}

• expires on Aug 22, 2026, 8:15 AM UTC-08:00

[github-actions]

This issue was automatically closed because it expired on 2026-08-22T16:15:28.191Z.

Closed by Workflow