Telemetry: synthetic cmd.provision/cmd.package spans always report Success under zd up (since 1.25.0); no cmd.deploy span emitted

[jongio]

Summary

Since v1.25.0 (PR #7776, the unified up-graph), azd up no longer reflects the real provision/package outcome in the cmd.provision and cmd.package telemetry spans, and no longer emits a cmd.deploy span at all. This makes cmd.provision success under azd up always report Success, silently inflating provision success-rate dashboards and deflating deploy success-rate dashboards. Root-caused in Azure/azure-dev-pr#1797.

Root cause (code)

cli/azd/internal/cmd/up_graph.go:162–175 starts synthetic cmd.package and cmd.provision spans and ends them with a plain .End():

_, provisionSpan := tracing.Start(ctx, "cmd.provision")
provisionSpan.SetAttributes(fields.CmdFlags.StringSlice(parentChangedFlags))
defer func() {
    usageAttrs := tracing.GetUsageAttributes()
    packageSpan.SetAttributes(usageAttrs...)
    packageSpan.End()          // status left Unset
    provisionSpan.SetAttributes(usageAttrs...)
    provisionSpan.End()        // status left Unset
}()

SetStatus / RecordError / MapError is never called on either span.

The AppInsights exporter derives success from span status:

// cli/azd/internal/telemetry/appinsights-exporter/span_to_envelope.go:58
requestData.Success = span.Status().Code != codes.Error

An Unset status is not codes.Error, so these spans always export Success = true / ResponseCode = "Success", regardless of the real outcome.

By contrast:

• Real graph steps record status correctly: span.EndWithStatus(stepErr) (cli/azd/pkg/exegraph/scheduler.go:458); the provision step is tagged ["provision"] (up_graph.go:825), deploy steps are deploy-<svc> / publish-<svc>.
• Real command spans get status from the telemetry middleware: on error it calls cmd.MapError(err, span) → span.SetStatus(codes.Error, code) (cli/azd/internal/cmd/errors.go:64–65), which also stamps the AppInsights ResultCode.
• Legacy azd up (≤1.24) ran provision/deploy/package as real child cobra commands via workflow.Runner.ExecuteContext, so each phase emitted a real, status-bearing cmd.* span.

Impact

1. cmd.provision success under azd up is pinned to Success. Since under-up invocations are a large share of all provisions, the aggregate cmd.provision success rate is inflated well above real reliability (root-cause analysis estimates the real rate is ~flat, while the reported metric jumped ~34 pts Apr→Jun 2026).
2. cmd.provision failure ResultCodes are lost for under-up runs. ARM/Bicep codes (service.arm.deployment.failed, tool.bicep.failed, etc.) never land on the synthetic span, so they vanish from the cmd.provision failure distribution and appear to "collapse."
3. cmd.deploy under azd up disappears entirely. No cmd.deploy span is emitted by the up-graph (there is zero tracing.Start(ctx, "cmd.deploy") in cli/azd). Deploy work runs as exegraph.step (deploy-<svc>). This drops the high-success under-up deploy population out of the cmd.deploy metric, deflating the deploy success rate.
4. cmd.package under azd up has the same always-Success problem as cmd.provision.

Proposed fix

Make the synthetic spans reflect the real outcome of the corresponding phase. In up_graph.go, capture per-phase errors from the graph result (steps are already tagged provision / deploy-* and prefixed package-*/publish-*/deploy-*) and:

• End cmd.provision with the aggregated provision-step error, running it through the same MapError path so status and ResultCode match a standalone azd provision.
• Do the same for cmd.package.
• Decide on cmd.deploy under up: either emit a synthetic cmd.deploy span carrying the aggregated deploy/publish-step status (restores dashboard continuity), or intentionally drop it and update the dashboards. Emitting it is preferable for parity with the legacy shape and with cmd.provision/cmd.package.

Alternative / interim: repoint provision and deploy dashboards at exegraph.step (tags provision / deploy-*) for the 1.25.0+ era, and treat the synthetic cmd.* spans as timing/shape only.

Acceptance criteria

• A failing provision under azd up produces a cmd.provision span with Success = false and the correct ResultCode (matches standalone azd provision).
• A failing deploy under azd up is represented with Success = false in whichever span the deploy dashboard consumes.
• cmd.package under azd up reflects the real package outcome.
• Telemetry test coverage (e.g. Test_CLI_Telemetry_NestedCommands) asserts error status propagation, not just span presence/shape.
• A short note for the telemetry/dashboard owner documenting the 1.25.0–<fix> window where cmd.provision/cmd.package over-report success and cmd.deploy under-up is missing, so historical dashboards can be corrected.

References

• Discussion: Azure/azure-dev-pr#1797
• Introduced by: feat(exegraph): graph-driven execution engine for up/provision/deploy #7776 (merge 618699082, first shipped in cli/azd/v1.25.0, 2026-05-08)
• Key code: cli/azd/internal/cmd/up_graph.go:162–175 and :825; cli/azd/internal/telemetry/appinsights-exporter/span_to_envelope.go:58; cli/azd/internal/tracing/tracer.go:108–122; cli/azd/internal/cmd/errors.go:48–65; cli/azd/pkg/exegraph/scheduler.go:458