Build analysis incorrectly marks build as green despite actual build break

[michalpavelka]

Summary

Build analysis is incorrectly classifying genuine build breaks as "Non-Deterministic Test Failures" and marking them green (flaky/passed-on-retry), even when no retry was actually attempted and the test cannot pass.

Reported By

• Michal Strehovsky — posted in .NET Core Eng Services Partners > First Responder channel
• Julie Lee — CC'd to team

Details

A build break occurred in dotnet/runtime main branch:

• CI correctly showed the build as red (failed)
• Build analysis incorrectly classified it as:
  • "Non-Deterministic Test Failures Detected"
  • "These tests passed after a retry (flaky tests)"
• No evidence of a retry attempt exists in Azure DevOps
• The last DevOps run that build analysis was based on shows red and build break
• Developers who introduced the test confirm the test could never pass

Suspicious Build Analysis Run

https://github.com/dotnet/runtime/runs/77478246586

Impact

• Developers may miss real build breaks if build analysis marks them green
• Undermines trust in the build analysis system
• Could allow broken code to flow downstream undetected

Expected Behavior

Build analysis should not classify a build as "green/flaky" when:

1. No retry was actually performed
2. The underlying AzDO run still shows failure
3. The test is deterministically failing (not flaky)