[mashkovd]

Summary

Opens the observability-eval sandbox for the #1280 tracing bake-off with one candidate, Grafana Tempo (owner decision, 2026-10-10). Teardown date: 2026-10-24.

• Tempo 2.10.8, single binary, 1 replica, from the community chart tempo 2.4.0 (https://grafana-community.github.io/helm-charts; chart to appVersion mapping checked against that repo's index.yaml, generated 2026-10-09). The old grafana/tempo-distributed pin is gone.

• Storage: Cloudflare R2 bucket tempo-traces through a bucket-scoped token. An ExternalSecret tempo-r2 reads Vault KV platform/r2-tempo through the existing vault-backend ClusterSecretStore. That store already reads secret/data/platform/*, so no Vault change is needed. Tempo expands ${TEMPO_S3_*} from the Secret (-config.expand-env=true), so the bucket, endpoint and keys never appear in git, and a render test fails if a literal shows up.

• WAL on an emptyDir with sizeLimit: 2Gi, no PVC. The quota now allows persistentvolumeclaims: "0" and requests.storage: "0". Retention is 336h (block_retention, enforced by the compactor), with the bucket's 30-day lifecycle rule as a backstop. max_block_duration is 5m so blocks reach R2 within minutes.

• Resources: requests 100m / 512Mi, limits 1 CPU / 1Gi, GOMEMLIMIT=800MiB. The chart's default 1 GiB memory ballast is turned off, because with it the GC would run continuously under GOMEMLIMIT. The quota is cut from 4 CPU / 12Gi requests to 1 CPU / 1536Mi.

• Collector fan-out: debug plus otlp/eval-tempo. Receivers, processors and the other pipelines are unchanged, and a test asserts it.

• Grafana datasource "Tempo (eval)" (uid tempo-eval). It is rendered only while the sandbox is open and the tempo candidate is enabled.

• NetworkPolicies for Tempo: :3200 from monitoring (Grafana, vmagent), and egress :443 to Cloudflare's published IPv4 ranges only. OTLP 4317/4318 from monitoring was already allowed by the namespace template.

• Scrape: a VMServiceScrape (job="tempo") on the non-headless Service only. The chart's ServiceMonitor stays off, which avoids the prometheus-pushgateway is scraped twice, so every pushed metric is duplicated and its alerts fire twice #1159 double scrape.

• Alerts (vm-rules/tempo-eval-alerts.yaml, promtool unit tests in vm-rules/tests/), routed to Telegram only by a TempoEval.* route:

  • TempoEvalNotScraped
  • TempoEvalMemoryNearLimit (RSS > 85% of the limit)
  • TempoEvalOOMKilled
  • TempoEvalRestarting
  • TempoEvalIngestRefused
  • TempoEvalDiscardedSpans
  • TempoEvalCollectorExportFailures
  • TempoEvalCollectorDroppedSpans
  • TempoEvalFlushFailures
  • TempoEvalObjectStorageErrors (non-2xx/404 from the bucket)
  • TempoEvalCompactionOrRetentionErrors

  No rule uses absent(). Each one is silent with the sandbox gone, and the tests prove this, including the scaled-to-zero case.

• Docs:

  • docs/runbooks/tracing-bake-off.md is rewritten for the Tempo-only round: capacity, gated rollout, accepted risks and teardown, including emptying the bucket.
  • ADR 0001 gets a "Bake-off scope amendment" that declares the soak window and volume before measuring. Its status stays Proposed, the verdict stays UNFILLED and the rubric stays frozen.
  • docs/runbooks/otel-collector.md describes the fan-out's current state.

• yamllint.yml ignores the new datasource template. It starts with a template comment, so the eval-off render emits no document, and yamllint cannot parse a file that starts that way. Same reason as the existing eval templates.

Are real producers forwarded? No, there are none yet.

The fan-out is ON as merged, and that is safe because no workload emits OTLP to the collector today. Checked on 2026-10-10:

• VictoriaMetrics has no otelcol_receiver_* series in 30 days.
• No pod has an OTEL_*ENDPOINT env.
• No base-service release sets otel.enabled.

So the only spans that reach Tempo are the ones an operator submits through otel-trace-fixture. Instrumenting a real producer is a separate change, recorded on #1280 as an amendment to the declared volume first.

Resources (measured read-only 2026-10-10)

Allocatable per worker is 7700m CPU and about 14.66GiB memory. The control plane is tainted NoSchedule. Tempo adds +100m / +512Mi requests and +1 CPU / +1Gi limits to whichever worker it lands on.

Node	CPU requests now → with Tempo	Memory requests now → with Tempo	Memory limits now → with Tempo	Memory used now / 7d max	CPU used now / 7d p95
ftx	3255m (42%) → 3355m (44%)	8902Mi (59%) → 9414Mi (62%)	114% → 121%	6.07 / 7.02 GiB	1.25 / 1.35 cores
hnn	2865m (37%) → 2965m (39%)	8359Mi (55%) → 8871Mi (59%)	112% → 119%	6.63 / 7.92 GiB	3.01 / 2.72 cores
xgf	3110m (40%) → 3210m (42%)	8995Mi (59%) → 9507Mi (63%)	134% → 141%	5.77 / 10.73 GiB	1.30 / 1.66 cores

Tempo's expected actual use at the declared synthetic volume is about 200–400MiB RSS and well under 0.1 core. The WAL emptyDir is capped at 2Gi of node disk, and each worker has about 116–122GiB free.

Risks (accepted for the bake-off only, listed in the ADR and the runbook)

• WAL loss on pod deletion or reschedule. A container restart or OOM kill keeps the emptyDir and the WAL is replayed. A pod move loses up to about one block window (5 min) of unflushed spans.
• An R2 outage longer than the 2Gi WAL can absorb loses spans. The kubelet evicts the pod at the sizeLimit. TempoEvalFlushFailures and TempoEvalObjectStorageErrors fire long before that.
• Single replica: a restart is an ingest gap. That is part of what the operations dimension measures.
• Retention cannot be observed inside a window of 14 days or less. The bucket lifecycle bounds storage.
• The Grafana datasource persists in Grafana's DB after teardown. The runbook deletes it explicitly.
• Expected one-off alert: TempoEvalCollectorExportFailures may fire once during the first sync, while the collector rolls before Tempo is Ready.

Rollout

1. Merge. This merge is the sandbox opening. Nothing is applied by hand.

2. Argo CD sync. Check that tempo-r2 is SecretSynced, tempo-0 is Ready and the quota status.used is within limits. The collector rolls with the new exporter.

3. Synthetic span proof: collector → Tempo → Grafana. Run argo submit -n argo-workflows --from clusterworkflowtemplate/otel-trace-fixture, then check:

   • otelcol_exporter_sent_spans{exporter="otlp/eval-tempo"} goes up;
   • the trace tree is complete in Grafana "Tempo (eval)";
   • tempo_ingester_blocks_flushed_total > 0 with failed_flushes flat.

   Record the timestamp on spike(observability): run the live tracing backend bake-off and decide #1280. It starts the declared soak window.

4. Real producers later, each as its own reviewed change, recorded on spike(observability): run the live tracing backend bake-off and decide #1280 before it lands.

A permanent backend and prod-wide tracing are separate owner gates after the ADR verdict.

Rollback

• Do not git revert this PR. Reverting templates while Argo CD holds the live otel-eval-tempo Application orphans it.
• Instead, use the one-commit teardown in docs/runbooks/tracing-bake-off.md §9: otelCollector.eval.enabled: false, the tempo candidate enabled: false, teardownAfter: "". Prune cascades through the Application finalizer and removes the namespace, Secret, policies, scrape, exporter and datasource ConfigMap.
• Then delete the "Tempo (eval)" datasource in Grafana, and empty the tempo-traces bucket if Tempo is not selected.
• Quick stop without a teardown: set the tempo candidate enabled: false. This removes the Application, the exporter and the datasource, and leaves the namespace.

Verification done locally

• helm lint passed for all charts.

• Full validate-manifests.yml validate job, run step by step: 47/48 steps passed, including:

  • kubeconform on the bootstrap render, the eval overlay and the raw manifests (with eval/tempo/);
  • check-vm-rules.sh (--selftest plus the real run, with the new promtool tests);
  • all render tests and validate-eval-teardown-date.py (--selftest plus the real run).

  The one failure is test_otel_collector_redaction_e2e.py. It needs a working local Docker daemon, which was broken in this environment, and this PR does not touch it. CI runs it.

• yamllint 1.35.1 with the workflow's relaxed config passed.

• Off-state check:

  • Rendering the full bootstrap with --set otelCollector.eval.enabled=false on origin/main and on this branch differs only by the TempoEval.* Alertmanager route.
  • The VMRule file in infra-components is always deployed but is silent without the sandbox, and the tests prove it.

• tests/test_otel_collector_backends_render.py was extended:

  • the golden/T5 baselines now render with an explicit eval-off overlay;
  • the committed default must equal the baseline plus exactly otlp/eval-tempo;
  • the Tempo pins are asserted literally (chart, revision, replicas, retention, GOMEMLIMIT, the emptyDir cap, no persistence, ${...} placeholders only, ServiceMonitor off);
  • the datasource host must match otlpEndpoint, and there is no datasource when the sandbox is off or tempo is disabled;
  • the quota storage caps are pinned to "0".

  Assertions were mutation-checked: revision, literal endpoint, sizeLimit and GOMEMLIMIT each turn the test red.

• Local e2e with the exact rendered Tempo config (Tempo 2.10.8, a local S3 emulator instead of R2, a read-only root filesystem):

  • the config parses and expand-env works;
  • OTLP ingest → search returns the traces;
  • a forced flush wrote the block to the bucket (blocks_flushed_total 1, PUT 200);
  • stopping the emulator produced failed_flushes_total and tempodb_backend_request_duration_seconds_count{status_code="500"}, which are the series the storage alerts use.

Not verified (needs the live cluster after merge)

• The real R2 token and endpoint shape in Vault. The ExternalSecret strips the scheme and trailing slash in either form.
• Egress to R2 through the Cloudflare-range policy.
• The Grafana sidecar picking up the datasource.
• vmagent scraping across namespaces.
• The otel-trace-fixture → collector path.

Refs #1280

🤖 Generated with Claude Code

https://claude.ai/code/session_013zqBqP9sSbVTd1kXRLA6AU

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[claude]

Claude finished @mashkovd's task in 4m 8s —— View job

---

Review complete ✅

• Gather context (diff, CLAUDE.md, changed files)
• Review Helm/ArgoCD correctness (templates, values, quota)
• Review security (secrets, RBAC, NetworkPolicy)
• Review operational safety (resources, alerts, health)
• Review test coverage
• Post inline findings (none needed)
• Submit verdict: APPROVE

No P1/P2 findings.

Checked against git diff 0b8a16cb744f87a7bb9ab18d885d17f0fcb8afec...f52d1add33d566c7a8b6d500821e130868f0e914:

• Secrets: tempo-r2 ExternalSecret follows the established ESO/Vault pattern (same shape as the Loki object-storage credentials); no literal bucket/endpoint/keys in git, and the rendered Tempo config only ever carries ${TEMPO_S3_*} placeholders (enforced by tests/test_otel_collector_backends_render.py).
• NetworkPolicy: scoped to the Tempo pod only (never podSelector: {}); egress CIDR list for Cloudflare's published IPv4 ranges is byte-identical to infrastructure/k3s-preview/extra-manifests/cloudflare-origin-allowlist.yaml.tpl (verified directly, all 15 ranges match).
• Quota/LimitRange: requests.cpu/memory and limits.cpu/memory leave consistent headroom for one rolling replacement of the single Tempo pod (100m/512Mi request, 1 CPU/1Gi limit); persistentvolumeclaims/requests.storage pinned to "0" matches the emptyDir-WAL/no-PVC design, and the LimitRange max (1536Mi) comfortably exceeds Tempo's own 1Gi limit.
• GOMEMLIMIT/ballast: 800MiB GOMEMLIMIT under the 1Gi container limit, with the chart's default 1GiB ballast turned off — consistent with the stated rationale (ballast counts as live heap against GOMEMLIMIT).
• Alerting: TempoEvalNotScraped's unless on () idiom correctly distinguishes "ready StatefulSet, no successful scrape" from both "scaled to zero" and "sandbox closed," matching the promtool unit tests (including the scaled-to-zero and closed-sandbox quiet cases). No rule uses absent(), as claimed. The TempoEval.* Alertmanager route is a prefix match placed ahead of the catch-all, routing to Telegram only.
• Render tests: the new committed-default assertions diff the committed render against the eval-off baseline and pin the traces pipeline's receivers/processors as unchanged, the Tempo chart/values literally (chart tempo 2.4.0, replicas 1, retention 336h, GOMEMLIMIT, emptyDir sizeLimit: 2Gi, no persistence, S3 placeholders, ServiceMonitor off), and that a disabled candidate emits no datasource.
• Docs: ADR amendment, runbook rewrite, and otel-collector.md update are internally consistent with the values/quota/resource numbers in the diff (capacity table math checks out against the stated Tempo footprint).

Could not independently verify the community tempo chart's exact values schema (nested tempo: vs top-level keys) against the live chart repo — this sandbox has no network/package access — but the PR description states this was validated via local e2e against the real rendered config (OTLP ingest → search, forced flush to a local S3 emulator, forced failure), and the internal consistency of the values block (e.g. extraVolumeMounts nested under tempo: vs extraVolumes/persistence/serviceMonitor at top level) matches the conventional Grafana chart pattern, so I did not flag it as a finding.

[github-actions]

Antigravity (agy) review — informational

Model: gemini-3.8-flash-medium · commit f52d1ad.

Findings

P1: Duplicate volume and volume mount in Tempo candidate values breaks StatefulSet admission

• File: platform-gitops/bootstrap/values.yaml (approx. line 184)
• Anchor: + extraVolumes:

Concrete failure scenario:
In the community tempo chart (2.4.0), when persistence.enabled: false is set, the chart's templates/statefulset.yaml automatically creates an emptyDir volume named tempo-data and mounts it to container path /var/tempo.

Because values.yaml also declares:

            extraVolumeMounts:
              - name: tempo-data
                mountPath: /var/tempo
          persistence:
            enabled: false
          extraVolumes:
            - name: tempo-data
              emptyDir:
                sizeLimit: 2Gi

the rendered StatefulSet ends up with two volumes named tempo-data in spec.template.spec.volumes and two mounts named tempo-data for /var/tempo in spec.template.spec.containers[0].volumeMounts.

The Kubernetes API server strictly rejects duplicate volume names and duplicate volume mounts at admission with validation errors (spec.template.spec.volumes[...].name: Duplicate value: "tempo-data" and spec.template.spec.containers[0].volumeMounts[...].name: Duplicate value: "tempo-data"). Consequently, when ArgoCD syncs the otel-eval-tempo Application, the StatefulSet creation will be rejected, and the tempo-0 pod will never be created or run. (Note: tests/test_otel_collector_backends_render.py only validates the values object in the Application manifest and does not render the upstream chart, so CI did not catch this.)

Proposed fix:
Remove the duplicate tempo-data declarations from extraVolumeMounts and extraVolumes in platform-gitops/bootstrap/values.yaml (letting the chart's native emptyDir handle /var/tempo when persistence.enabled: false), and update the corresponding assertions in tests/test_otel_collector_backends_render.py.

[claude]

No P1/P2 findings (0 P3). Tempo values (single binary, S3/R2 storage, WAL emptyDir, GOMEMLIMIT), quota/LimitRange sizing, NetworkPolicy Cloudflare egress CIDRs (verified byte-identical to infrastructure/k3s-preview's allowlist), ExternalSecret/Vault wiring, alert routing, and render tests all check out. Good to merge.

[mashkovd]

Re the agy P1 ("duplicate tempo-data volume and mount"): refuted, no change. The community tempo 2.4.0 chart does not create any volume for /var/tempo when persistence.enabled: false.

templates/statefulset.yaml in the 2.4.0 tarball (digest fad57f9d…):

        volumeMounts:
        {{- if .Values.tempo.extraVolumeMounts -}}
          {{ toYaml .Values.tempo.extraVolumeMounts | nindent 8 }}
        {{- end }}
        - mountPath: /conf
          name: tempo-conf
        {{- if .Values.persistence.enabled }}
        - mountPath: /var/tempo
          name: storage
        {{- end }}
...
      volumes:
      {{- if .Values.extraVolumes -}}
        {{ toYaml .Values.extraVolumes | nindent 6 }}
      {{- end }}
      ...
      - configMap:
          name: {{ template "tempo.name" . }}
        name: tempo-conf

• The chart's only /var/tempo mount is storage, and it is gated on persistence.enabled. With persistence off there is no chart-side volume and no chart-side mount.
• There is no tempo-data anywhere in the chart. It exists only because these values add it.

helm template otel-eval-tempo tempo-2.4.0.tgz -f <the valuesObject this PR renders> produces exactly one volume and one mount per name:

      volumes:
      - emptyDir:
          sizeLimit: 2Gi
        name: tempo-data
      - configMap:
          name: tempo
        name: tempo-conf
# container volumeMounts: tempo-data -> /var/tempo, tempo-conf -> /conf
# volumeClaimTemplates: none

Removing extraVolumes / extraVolumeMounts as proposed would actually break the pod. /var/tempo would then be on the container's readOnlyRootFilesystem: true, so the WAL could not be written and the 2Gi cap would be gone.

I also ran the same rendered config locally under Tempo 2.10.8 with a read-only root filesystem and the WAL directory mounted separately. It became ready, ingested OTLP, returned the traces from search and flushed a block to an S3 emulator.