Skip to content

fix(ci): Prevent duplicate recurring CI failure issues - #19804

Closed
Ankit Jain (radical) wants to merge 10 commits into
mainfrom
radical-ankj/canonical-failure-causes
Closed

Ankit Jain (radical) wants to merge 10 commits into
mainfrom
radical-ankj/canonical-failure-causes

Conversation

@radical

@radical Ankit Jain (radical) commented Aug 29, 2026 •

Copy link
Copy Markdown
Member

[automated] Recurring CI failures could split across multiple memory records and GitHub issues. Repeated occurrences such as:

Process completed with exit code -1073741502 (0xC0000142)

could receive different proposed cause IDs, lose a historical issue association, or mutate duplicate issues before the canonical issue was updated safely.

Root cause: Agent-proposed IDs were treated as stable identities before comparison with historical records, while issue reconciliation and publication receipts were distributed across producer-specific workflow steps. Same-test records, aliases, trusted job evidence, labels, and occurrence history could therefore diverge across runs.

The fix: Resolve proposed causes deterministically against canonical IDs, transitive aliases, normalized test names, trusted retry patterns, explicit matchers, and trusted failed-job attribution before persistence. Compatible historical same-test records converge on the oldest canonical root, while fresh proposals reuse an existing family only when authoritative identity evidence supports it.

Trusted retry patterns and matchers can redirect the current proposal, but they rewrite an existing historical root only when the same trusted diagnostic signal also matches that root. This preserves intentional split-family convergence without allowing an unrelated proposed ID to absorb another issue family or inherit its aliases and issue URL.

Stored-alias proposals follow the same trusted resolution path. A redirect changes only the current occurrence unless the alias family shares the trusted diagnostic signature, and ambiguous trusted matches fail closed.

Trusted failed-job attribution now enforces non-empty, unique, positive job IDs at the shared validation boundary before any rerun side effect.

The shared tracking-issue.js planner/executor owns exact-marker lookup, oldest-canonical selection, post-create relisting, comment hydration, reopening, label repair, and optional duplicate closure. Canonical updates execute before duplicate comments or closes, and missing cause labels are repaired without reopening a closed receipt-only issue.

Cause publication preserves a bounded rolling occurrence section and consolidates valid rows from open and closed aliases before closing open duplicates. Malformed or conflicting open history suppresses duplicate reconciliation; malformed closed history is skipped with a warning so it cannot permanently block cleanup. Replay-only history consolidation does not reopen a closed canonical issue, while a genuinely new occurrence still does.

Publication receipts are bound to the selected canonical issue URL and resolved across canonical and transitive alias records. Replaying a trimmed occurrence remains idempotent without treating memory persistence before a failed issue mutation as successful publication.

Main-branch breakage titles and diagnostics are rendered from trusted run context. Agent-derived fields are rendered as literal Markdown, occurrence delimiters and rows are recognized only as complete standalone lines, and retry-pattern output matching uses trusted bounded job-log tails rather than agent-authored evidence.

Validation: 554 focused tests passed across AgenticWorkflowTests, AnalyzeCiFailureCauseIssuesTests, AnalyzeCiFailureCauseResolverTests, AnalyzeCiFailureWorkflowTests, AutoRerunTransientCiFailuresTests, and TrackingIssueTests. JavaScript syntax checks, focused formatting, git diff --check, and gh aw compile analyze-ci-failure --validate --actionlint --shellcheck also passed.

Fixes #19578

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 19804

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 19804"

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds deterministic CI-failure cause resolution to prevent duplicate memory records/issues and improve job attribution.

Changes:

  • Canonicalizes proposed cause IDs using prior causes, test names, aliases, retry patterns, and matchers.
  • Serializes publication and records per-cause job associations.
  • Adds resolver, workflow, and retry-pattern coverage.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
.github/workflows/analyze-ci-failure-cause-resolver.js Implements cause canonicalization and attribution.
.github/workflows/analyze-ci-failure.md Integrates resolver and serializes publication.
.github/workflows/analyze-ci-failure.lock.yml Updates generated workflow.
.github/workflows/auto-rerun-transient-ci-failures.js Validates optional canonical cause IDs.
eng/test-retry-patterns.json Assigns the Windows initialization cause ID.
docs/ci/auto-rerun-transient-ci-failures.md Documents causeId usage.
AnalyzeCiFailureCauseResolverTests.cs Adds resolver and workflow tests.
analyze-ci-failure-cause-resolver.harness.js Provides the Node test harness.
AutoRerunTransientCiFailuresTests.cs Tests retry-pattern validation.

Comment thread .github/workflows/analyze-ci-failure.md Outdated
Comment thread .github/workflows/analyze-ci-failure-cause-resolver.js Outdated
Comment thread .github/workflows/analyze-ci-failure-cause-resolver.js Outdated
Copilot AI review requested due to automatic review settings August 29, 2026 20:39
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Comment thread .github/workflows/analyze-ci-failure-cause-resolver.js
Comment thread .github/workflows/auto-rerun-transient-ci-failures.js
Comment thread .github/workflows/analyze-ci-failure-cause-resolver.js Outdated
Copilot AI review requested due to automatic review settings August 30, 2026 03:17
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Comment thread .github/workflows/analyze-ci-failure-cause-issues.js Outdated
Copilot AI review requested due to automatic review settings August 30, 2026 03:43
@radical
Ankit Jain (radical) force-pushed the radical-ankj/canonical-failure-causes branch from d591fdc to 68cf3ee Compare August 30, 2026 03:43
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

.github/workflows/analyze-ci-failure-cause-resolver.js:416

  • This combines output evidence from all jobs with jobNames.some(...), so a two-field job pattern can match across different jobs. For example, a cause spanning Windows and Linux is canonicalized by a { jobName: "windows", output: "LINUX_ONLY_TOKEN" } rule when only the Linux test contains that token. Job-level retry rules require both matchers to hold for the same job; build evidence per job and accept a rule only when one job satisfies all of its matchers, otherwise unrelated causes can be merged under one canonical ID.
        .filter(pattern => !pattern.output || matchesConfiguredPattern(pattern.output, evidence))
        .filter(pattern => !pattern.jobName || jobNames.some(jobName => matchesConfiguredPattern(pattern.jobName, jobName)))

Comment thread .github/workflows/analyze-ci-failure-cause-resolver.js Outdated
Copilot AI review requested due to automatic review settings August 30, 2026 05:17
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 2 comments.

Comment thread .github/workflows/analyze-ci-failure-cause-resolver.js Outdated
Comment thread .github/workflows/tracking-issue.js Outdated
Copilot AI review requested due to automatic review settings August 30, 2026 05:56
Explain how canonical cause IDs and aliases unify repeated CI analyses, how
retry patterns provide stable mappings, and how occurrence receipts prevent
duplicate publication.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68afc81c-eaf6-4e13-a0e8-9528d8824b0c

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Alias/test mismatches can merge unrelated flaky tests, normalization differs across validation boundaries, and an obsolete renderer remains checked in.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 High severity · 1 Medium severity · 1 Low severity

Open (4)

Comment thread .github/workflows/analyze-ci-failure-cause-resolver.js
Comment thread .github/workflows/analyze-ci-failure-validation.sh
@github-actions

This comment has been minimized.

Occurrence receipts were tracked only by workflow run ID. When canonical
issue selection changed across aliases, a receipt from the previous issue
could suppress the canonical update before duplicate reconciliation closed
the issue that still contained the occurrence.

Persist the selected issue URL with each receipt and honor the receipt only
for that issue. Legacy URL-less receipts and count-based inference remain
accepted when exactly one matching issue exists, while multi-issue
reconciliation always verifies the selected canonical body.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68afc81c-eaf6-4e13-a0e8-9528d8824b0c

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Stored aliases are not resolved as identities, and existing canonical issues can lose required label reconciliation.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 High severity · 2 Medium severity

Open (4)
Resolved since last review (2)

Comment thread .github/workflows/analyze-ci-failure-cause-resolver.js
Comment thread .github/workflows/analyze-ci-failure-cause-issues.js
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Existing main-breakage issues are migrated without attaching the required main-ci-break label.

Review effort: Balanced
Findings: 2 High severity · 2 Medium severity

Open (4)

Recurring CI failure reconciliation could split canonical histories when
a proposed flaky ID reused another test's alias, when stored aliases were
not indexed, or when JavaScript and jq normalized Unicode test names
differently. Existing canonical issues could also permanently lose their
cause-specific labels because labels were only applied during creation.

Validate named flaky families against stored test identity, resolve stored
aliases through their canonical records with fail-closed collision checks,
and align test-name folding with jq's ASCII semantics. Route supplemental
label repair through the shared tracking issue engine without reopening
closed receipt-only issues.

Regression coverage exercises mismatched aliases and roots, Unicode test
names, direct and retry-pattern alias reuse, ambiguous aliases, cross-type
collisions, label repair, and closed-issue behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68afc81c-eaf6-4e13-a0e8-9528d8824b0c
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The default recordRun path can reopen a canonical issue without recording the run when its marker exists only on another duplicate.

Review effort: Balanced
Findings: None

Resolved since last review (4)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Preserve canonical failure comments when run marker is on duplicate

.github/​workflows/​tracking-issue.js:469

When closeDuplicates is false (the default used by the existing recordRun callers), this now deduplicates against every planning match. If the run marker exists only on a noncanonical duplicate, the selected canonical receives no failure comment; when all matches are closed it is even reopened with no occurrence recorded there. Since duplicate reconciliation is disabled, there is also no link to the issue that contains the run. Preserve the old behavior by treating a marker on another issue as a reason to comment on the selected canonical.

The workflow split analysis publishing and PR commenting into separate
steps, but the contract test still searched for the old combined step
names. CI therefore failed before verifying the intended artifact wiring.

Assert the current publisher and comment step names while retaining all
existing environment and script checks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68afc81c-eaf6-4e13-a0e8-9528d8824b0c
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Duplicate reconciliation can close an alias issue before transferring its unique occurrence rows to the canonical issue.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity

Open (1)

Comment thread .github/workflows/analyze-ci-failure-cause-issues.js Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Canonical issue reconciliation rebuilt occurrence history from only the
selected issue before closing aliases. Distinct rows on an alias therefore
disappeared from the canonical view, including replays after the current
run already had a publication receipt.

Merge validated occurrence rows from open aliases into the canonical
update, deduplicate identical run rows, and retain deterministic ordering
and bounded totals. If alias history is malformed or conflicts, publish
the current occurrence but leave duplicates open.

Regression coverage includes new publication, receipt replay, malformed
history, and conflicting rows.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68afc81c-eaf6-4e13-a0e8-9528d8824b0c
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The resolver can merge an unrelated historical cause when trusted evidence redirects an incorrect existing proposal ID.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity

Open (1)
Resolved since last review (1)

Comment thread .github/workflows/analyze-ci-failure-cause-resolver.js Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Closed duplicate issues can contain newer occurrence rows that are omitted from the canonical issue’s claimed “most recent” history.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity

Open (2)

Comment thread .github/workflows/analyze-ci-failure-cause-issues.js Outdated
Trusted retry patterns and matchers could redirect an agent proposal that
named an unrelated historical root, permanently aliasing separate issue
families. Closed aliases were also excluded from occurrence consolidation,
leaving canonical issue history incomplete.

Require the trusted identity signal to match the proposed historical root
before rewriting it. Import valid occurrence rows from closed aliases while
keeping replay-only closed canonicals closed and ignoring malformed closed
history without blocking open duplicate reconciliation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68afc81c-eaf6-4e13-a0e8-9528d8824b0c
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The rerun validator now accepts duplicate job_ids, weakening the attribution boundary before rerun side effects.

Review effort: Balanced
Findings: None

Resolved since last review (2)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Reject duplicate job IDs in shared rerun validator

.github/​workflows/​analyze-ci-failure-cause-resolver.js:1007

The shared validator now accepts duplicate job_ids (for example, [1, 1]), even though the rerun contract requires them to be unique and the replaced inline validator explicitly enforced that invariant. Because the rerun job calls this function directly on agent output, malformed attribution can now pass validation and reach the rerun side effect. Add a set-size check here.

The shared cause-attribution boundary accepted duplicate and non-positive
job IDs even though the workflow contract requires unique positive IDs.
That allowed malformed cause data to reach the rerun path after replacing
the workflow's inline validation.

Validate the complete job ID contract before attribution or rerun side
effects, and cover both the shared resolver and compiled rerun workflow.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68afc81c-eaf6-4e13-a0e8-9528d8824b0c
@radical

Copy link
Copy Markdown
Member Author

[automated] Fixed by restoring unique, positive job ID validation in the shared attribution boundary before rerun side effects. The direct resolver boundary and the compiled rerun workflow now both reject duplicate and non-positive IDs.

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Stored-alias proposals currently bypass trusted retry-pattern and matcher resolution, allowing failures to enter an unrelated cause family.

Review effort: Balanced
Findings: None

Previously missed (1)

In code that hasn't changed since last review

Medium severity Alias proposals bypass trusted resolution and enable misclassification

.github/​workflows/​analyze-ci-failure-cause-resolver.js:92

This skips trusted retry-pattern and explicit-matcher resolution whenever the agent proposes a stored alias. For example, if legacy-dns aliases dns-outage but the trusted job log matches the configured windows-init pattern, the same failure resolves to DNS when proposed as legacy-dns but to Windows when proposed as dns-outage. That lets an incorrect alias contaminate an unrelated historical family despite the trusted identity signal. Evaluate the trusted mechanisms for alias proposals too, then either redirect the current proposal or fail closed when the authoritative identities conflict.

Stored cause aliases bypassed trusted retry-pattern and explicit-matcher
resolution, so identical failure evidence could resolve to different cause
families depending on which alias the agent proposed.

Apply trusted current evidence consistently to alias proposals while keeping
unrelated alias families separate unless the same diagnostic signature
matches their stored history. Ambiguous trusted matches continue to fail
closed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 68afc81c-eaf6-4e13-a0e8-9528d8824b0c
@radical

Copy link
Copy Markdown
Member Author

[automated] Fixed in 998e6ef by applying trusted retry-pattern and explicit-matcher resolution consistently to stored-alias proposals. Trusted evidence can redirect the current occurrence without transferring an unrelated alias family; compatible families still converge only when the same diagnostic signature matches stored history, and ambiguous trusted matches fail closed.

@github-actions

Copy link
Copy Markdown
Contributor

Tests selector

1 / 99 PR test projects · 0 PR jobs, from 18 changed files.

Selected PR test projects (1 / 99)

Infrastructure.Tests

Selected PR jobs (0)

none


How these were chosen — grouped by what changed

📄 .github/workflows/analyze-ci-failure-cause-issues.js (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure-cause-resolver.js (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure-persistence.sh (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure-validation.sh (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure.lock.yml (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/analyze-ci-failure.md (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/auto-rerun-transient-ci-failures.js (changed)
→ 1 directly: Infrastructure.Tests

📄 .github/workflows/tracking-issue.js (changed)
→ 1 directly: Infrastructure.Tests

📄 eng/test-retry-patterns.json (changed)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/AgenticWorkflowTests.cs (changed test)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/AnalyzeCiFailureCauseIssuesTests.cs (changed test)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/AnalyzeCiFailureCauseResolverTests.cs (changed test)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/AnalyzeCiFailureWorkflowTests.cs (changed test)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/AutoRerunTransientCiFailuresTests.cs (changed test)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/TrackingIssueTests.cs (changed test)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/analyze-ci-failure-cause-issues.harness.js (changed test)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/analyze-ci-failure-cause-resolver.harness.js (changed test)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/WorkflowScripts/tracking-issue.harness.js (changed test)
→ 1 directly: Infrastructure.Tests

Job reasons

none


Selection computed for commit 998e6ef.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The broad identity, persistence, and GitHub issue lifecycle changes warrant final human review despite extensive focused coverage.

Review effort: Balanced
Findings: None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[automated] Analyze CI Failure opens duplicates when prior-cause context is truncated

2 participants