You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
CI failure tracking treated the test that happened to fail as the durable
identity of the failure. That breaks when one mechanism affects several tests,
or later appears under a different test or job.
Before: failing test name → cause ID → tracking issue
After: failure mechanism → stable cause ID → tracking issue
exact {test name, numeric job ID} → evidence for each occurrence
Tests and jobs now describe where a failure was observed. The underlying
mechanism identifies the long-lived cause and issue.
What broke
#19455 correctly made CI attribution and publication fail closed, but left one
scalar test_name serving as both cause identity and display text.
That model could not represent two valid cases:
Several tests fail because of the same underlying mechanism.
The same mechanism recurs later under a different test or job.
When analysis grouped several browser-debugger timeouts, publication looked for
one test with the combined display name and stopped:
Flaky-test cause must reference a validated flaky test
The run received neither its expected issue occurrence nor its analysis
comment. A later recurrence under another test could also create a duplicate
cause issue instead of extending the existing one.
Why this model
Decision
Reason
A cause ID represents the failure mechanism.
Test and job names can change between occurrences, but the issue should remain attached to the same underlying problem.
Each occurrence stores exact test and numeric job evidence.
Publication can authenticate what actually failed without turning that observation into permanent identity.
The analyzer decides whether diagnostics match an existing mechanism.
An exact fingerprint would split legitimate recurrences; fuzzy publisher matching could merge unrelated failures.
Persisted memory owns occurrence history.
Issue text is editable and retries can happen after partial failures, so publication must be reproducible from trusted stored evidence.
The publisher therefore verifies trusted evidence, unique ownership, and stored
identity. It does not guess whether two similar failures share a root cause.
Legacy data
Existing cause history is preserved rather than rewritten into the new
shape.
Stored cause IDs, diagnostics, issue links, occurrence dates, and
operator-authored notes remain intact.
Legacy records without an attempt retain their run-only link until trusted
replay evidence identifies the attempt. Known attempts use exact /attempts/N links, including attempt 1, so later retries cannot change what
a historical row points to.
Historical test_name and job labels remain descriptive evidence. They are
not promoted into new verified identity or assigned invented numeric job IDs.
Historical rows can recover a missing main scope only from the matching
trusted run summary. Otherwise missing context renders as unavailable;
editable issue text is not used to guess it.
Authenticated main scope wins over stale legacy PR metadata. Invalid cached
contexts such as #0 are rebuilt rather than preserved.
Older issue tables migrate on the next publication. When old data lacks the
new display fields, rows are derived deterministically from stored legacy
evidence, never from editable issue text or fresh analyzer output.
Ambiguous legacy issue layouts fail closed rather than risking deletion of
human-authored content.
This keeps existing tracking issues useful while allowing future occurrences to
use the stronger evidence model.
Issue history and recovery
The issue body is a projection, not a second occurrence database. Rebuilding it
from persisted history means retries can repair stale, edited, missing,
duplicate, or previously trimmed rows without changing the underlying evidence.
Text outside the managed occurrence section remains operator-owned. Pull-request
code failures also remain PR-comment-only; this change does not turn them into
persistent cause issues.
Regression coverage
The 427 passing workflow tests cover:
mechanism grouping, distinct-cause isolation, and exact test/job ownership;
legacy enrichment, immutable replay, partial-publication recovery, and issue
recreation from stored history;
exact attempt links, trusted historical scope recovery, bounded rendering,
operator-note preservation, and deterministic fallback for malformed legacy
context.
The generated workflow was regenerated with the repository-pinned gh-aw v0.88.7
compiler. No action pin changed.
[automated] Addressed the scalar-form occurrence-row finding from Copilot review in cc5bf4c. When a current-run flaky cause uses the supported test_name format, the new issue row now displays that verified test under each trusted job. Historical labels are still not used as evidence. Added a two-job regression test.
cause-job-names is called on the current cause file before add-occurrence, but this expression only reads .tests. The validator still accepts legacy flaky files without tests (the fallback at validation lines 503-504), and add-occurrence converts their test_name/job_ids into verified occurrence tests; therefore a valid legacy cause produces an issue occurrence row with only the job name and drops the test name. Build the legacy $tests fallback here as well so all issue rows show the verified tests.
Update legacy issue rows when enriching replay occurrences
.github/workflows/analyze-ci-failure.md:1006
When an attempt-1 replay enriches a legacy occurrence, OCCURRENCE_URL is the same base URL that the old run-only row already contains. This branch therefore skips the issue edit, so the persistent occurrence gains the verified tests but the linked issue continues to show the stale row without that evidence. Treat a matching legacy row as an enrichment/update (while still skipping an already-modern row), and add a publication regression test for this path.
[automated] Addressed the legacy issue-row migration finding in d87e8f1. Legacy attempt-1 enrichment now persists a refresh marker until the stale managed issue row is successfully replaced or a new issue is created. Replay reconstructs the row from persisted observations, preserves the original observation time, avoids duplicate edits when already current, retains the marker on unsupported bodies, and keeps modern occurrences immutable. Added end-to-end lifecycle and maximum-boundary coverage.
Only the legacy issue_row_needs_refresh path switches SOURCE_CAUSE_FILE to stored occurrence data. For an already-modern occurrence, merge-cause deliberately ignores a replay, but if memory was pushed and issue publication failed before its row existed, this default renders from the replay's CAUSE_FILE observations and date. A changed agent result can therefore publish an issue row that disagrees with the immutable stored occurrence; select the matching stored {run_id, run_attempt} occurrence for every replay when building display data, and add a missing-row regression.
[automated] Addressed the replay-publication inconsistency in aa07555e492.
Publication now reads the matching stored occurrence's date, URL, context, test evidence, and persisted job ordering. A retry after memory has been pushed can no longer render a different issue row for the same immutable run attempt. Historical non-flaky occurrences without numeric job IDs retain a safe fallback, and the replay/migration contracts cover both missing and existing managed rows.
The same commit disables gh-aw's per-run failed custom-job issue reporter for Analyze CI Failure and PR Documentation Check. This prevents duplicate [aw] Failed jobs issues such as #20352 and #20353 while retaining gh-aw's category-based agent failure reporting.
[automated] Followed up on the latest Copilot review overview in 20fe33cad5d.
Issue recreation now retains the authoritative historical grouped job label after the refresh marker is cleared.
Legacy enrichment preserves valid stored PR context and run scope.
Stored run ID, attempt, and URL must identify the same attempt. Only an absent legacy attempt defaults to attempt 1; malformed explicit values fail before merge persistence.
The live memory branch contains 143 cause files and 471 occurrences; all existing rows remain compatible with the stricter validation. The full workflow contract suite passes: 391 tests.
The fallback has the same jq type-coercion bug: a legacy pr_number stored as the string "41" satisfies > 0 and is emitted as a valid-looking #41 context instead of unavailable. This lets malformed persisted context survive issue-row reconstruction; validate the JSON type and integer range before rendering it.
[automated] Addressed the latest Copilot review finding in 4576a2abf54.
Legacy PR context now accepts only positive integral JSON numbers in occurrence
backfill, stored-row reconstruction, and replay publication. Strings, nulls,
booleans, zero, negatives, fractions, and collection values are treated as
missing context, preserving the authenticated main or unavailable fallback
instead of rendering a valid-looking PR reference or blocking publication.
Positive and negative coverage now exercises all three paths. The workflow
contract suite passes: 409 tests.
CI failure issues previously treated a test label as durable cause
identity, so grouped failures and legacy issue rows could be split,
overwritten, or reconstructed from editable issue text.
Persist stable cause IDs separately from occurrence evidence, rebuild
managed rows from authenticated memory, validate canonical run-attempt
URLs, and preserve verified legacy display and context metadata during
replay.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add contract coverage for grouped failures, canonical occurrence
validation, immutable replay metadata, legacy row recovery, malformed
persisted data, and failure-side effects.
The positive and negative cases verify that valid history is preserved
while untrusted or ambiguous data fails closed.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Document the distinction between durable cause identity and occurrence
evidence, the persisted-memory trust boundary, canonical history format,
and legacy recovery and fallback behavior.
This captures the invariants future workflow changes must preserve.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🔵 Needs a closer look
The extensive persistence, legacy migration, and external issue-publication changes warrant final human review despite comprehensive regression coverage.
[automated] Extended CI confidence requires action.
Reviewer actions
Required before merge: run the final Analyze CI Failure workflow in a safe trial repository. Done when: the run uses PR head e36b4fed7e45af370e19335e1389dcb241a940ad, groups multiple tests under one cause, replays stored history correctly, rejects mismatched evidence, and records the run link on fix(ci): Publish multi-test flaky causes to one issue #20333.
Ready-to-use pr-testing prompt
Use the pr-testing skill to test PR #20333. Focus on representative execution of the final Analyze CI Failure agentic workflow in a safe trial repository. Exercise one failure mechanism observed in multiple tests/jobs, immutable replay and legacy migration, and a negative case with untrusted or mismatched test/job evidence. Verify the published safe outputs create or update one cause issue, preserve stored occurrence history, and fail closed for invalid evidence. Do not mutate microsoft/aspire.
Change classification
Agentic CI workload:.github/workflows/analyze-ci-failure.md and its generated lock change analyzer instructions and publication behavior.
Workflow persistence and publication: three shell helpers now authenticate multi-test/job observations, preserve legacy history, and rebuild managed issue rows.
🔴 Agentic workflow execution — No representative final-head run
Why it applies: The changed agent instructions decide cause identity, associate trusted test/job evidence, and drive issue comments, issue creation, and memory-branch persistence. These are load-bearing mutation and fail-closed behaviors.
Evidence: No Analyze CI Failure workflow run exists for the PR branch or final head. Its workflow_run trigger is restricted to main, so normal PR CI cannot execute the changed workflow.
Gap or disposition: The tests validate scripts, fixtures, schemas, and generated wiring, but do not prove that the agent makes the intended semantic grouping decision or emits the correct safe output.
Action: Run the trial-repository scenario described above. The upstream workflow has real issue and branch side effects and should not be dispatched merely for validation.
🟢 Workflow contract and script tests — Final-head coverage passed
Why it applies: The PR changes validation, persistence, issue rendering, replay, and legacy migration scripts.
Evidence: Final-head CI run 36024774769 tested SHA e36b4fed7e45af370e19335e1389dcb241a940ad.
Observed result: All 1,465Infrastructure.Tests cases passed, including all 427/427AnalyzeCiFailureWorkflowTests cases. The downloaded TRX reports zero failures.
Coverage: Includes multi-test ownership, distinct-cause isolation, immutable replay, issue recreation, legacy migration, malformed rows and URLs, operator-content preservation, and invalid evidence rejection.
🟢 Agentic workflow compilation — Final source and lock are synchronized
Why it applies: Both the gh-aw Markdown source and generated .lock.yml changed.
Evidence: In a disposable checkout at the exact PR head, repository-pinned gh aw v0.88.7 successfully ran gh aw compile --validate and gh aw compile analyze-ci-failure.
Observed result: Recompilation produced an empty lock-file diff and a clean working tree.
Disposition: Compiler warnings were pre-existing advisory checks; none indicated lock drift or compilation failure.
🟢 CI graph, gating, and dependency pins — No silent coverage loss found
Why it applies: Changed helpers feed the Analyze CI Failure workflow and its issue/memory publication path.
Evidence: Consumer tracing found the changed helpers are consumed by the workflow source/generated lock and their Infrastructure test harnesses. No artifact name, downstream workflow contract, trigger, permission, if:, needs:, or continue-on-error change was introduced.
Coverage-loss audit: The workflow remains main-only for automatic workflow_run execution and retains manual workflow_dispatch; this PR did not narrow existing coverage.
Action pins: No uses: pin changed against the PR merge base, so no new action-provenance or allow-list validation is required.
⚪ Internal pipeline, outerloop, and deployment validation — Not applicable
Evidence: The diff does not change or feed internal Azure DevOps build/signing/publishing infrastructure, outerloop tests or execution plumbing, or deployment E2E scenarios and runtime inputs.
Disposition: No definition-1602, outerloop, or deployment run is required for this PR.
Tested Source: Detached worktree at the exact PR head
Live Workflow: PR-head analyze-ci-failure.md, recompiled with gh-aw v0.88.7
Trial-only adaptation: Changed the repository-owner activation gate from microsoft to radical; the analysis, validation, persistence, issue, and
safe-output logic was unchanged.
Result: ✅ Compilation, actionlint, and shellcheck passed. Recompilation
produced no lock-file diff. The compiler emitted only existing advisory
warnings about the shared dispatch conclusion concurrency group and actions/checkout@v6.0.3.
Test Scenarios Executed
Scenario 1: One mechanism across multiple tests and jobs
Objective: Verify three failed tests in two jobs with the same browser
debugger timeout are grouped into one recurring cause and one cause issue.
Coverage Type: Happy path / representative live execution
Exactly one cause record: browser-debugger-timeout
The issue and stored occurrence show both jobs and all three verified tests.
Trial fixture note: The first automatic analysis run https://github.com/radical/aspire-ci-failure-trial-20333/actions/runs/36190146998
reached the final publisher but failed because the empty trial repository did
not yet contain the expected test-failure label. Adding that repository
fixture and replaying the same immutable CI attempt completed successfully.
This was a trial-repository setup gap, not a PR failure.
Scenario 2: Immutable replay
Objective: Replay the same run and attempt after the occurrence had already
been persisted.
Coverage Type: Replay / idempotency
Status: ✅ Passed
Steps:
The first publisher attempt persisted run 36190131954, attempt 1.
The workflow was manually dispatched again for run 36190131954.
The replay completed successfully.
Result:
The stored cause still contained exactly one occurrence for (run_id=36190131954, run_attempt=1).
The occurrence retained its attempt-specific URL, trusted job IDs, tests,
issue table projection, and main context.
The cause issue was created from the stored occurrence without duplicating
history.
Scenario 3: Legacy migration and occurrence preservation
Objective: Verify the publisher migrates a legacy occurrence table while
retaining memory-authoritative history and human operator notes.
An issue body with the old PR occurrence-table header and no managed
occurrence markers
An operator note: [automated] Preserve this trial operator note during legacy migration.
Result:
The cause issue was updated in place; no second cause issue was created.
The legacy table was replaced by the canonical managed occurrence section.
All three histories were retained:
Seeded legacy occurrence
Original immutable occurrence
New CI occurrence
The operator note was preserved unchanged.
Modern occurrences retained exact trusted {test, job_id} evidence.
The legacy occurrence remained explicitly unavailable for context rather
than being reconstructed from editable issue text.
Scenario 4: Mismatched and untrusted test/job evidence
Objective: Verify invalid agent-supplied cause observations fail before
publication or persistence.
Coverage Type: Negative / trust boundary
Status: ✅ Passed
Cases:
Rebound the real Trial.Tests.FirstTimeout test from its trusted job to the
other failed job's ID.
Added an untrusted Attacker.Tests.Injected observation to a real failed
job ID.
Result:
Both exact PR-head validator executions exited 1 with:
::error::Cause browser-debugger-timeout.json references an unknown or incompatible failed job or test
The trial repository state was unchanged after both attempts:
Memory branch remained at edf18f8911521842fc1900e1056af1082167e30e
Cause issue updatedAt remained 2026-09-25T21:30:54Z
Cause-issue count remained exactly 1
CI Infrastructure Validation
What runs on the Aspire PR: The focused AnalyzeCiFailureWorkflowTests class ran in PR CI and was also rerun locally
against the exact PR head.
Manual/live validation: The gh-aw workflow was compiled and executed in a
private trial repository because the production workflow is intentionally
owner-gated and performs real issue and memory-branch mutations. No workflow or
issue was created in microsoft/aspire.
Results validation: Verified the final publisher's observable outputs, not
only run status:
Trusted TRX artifacts were selected by job/artifact naming contract.
The agent emitted one cause covering all exact test/job pairs.
Safe-output validation passed for valid evidence.
The publisher created and updated one cause issue.
Memory retained immutable and legacy occurrence history.
Invalid evidence failed before any remote mutation.
Dependency graph: Synthetic CI push failure → workflow_run activation →
trusted collection and artifact extraction → agent analysis → safe-output
validation → memory branch push → cause issue create/update.
gh-aw:compile --validate --actionlint --shellcheck passed and the PR lock
file regenerated without drift.
Failure-mode scan:
Repository-owner gate was adapted only in the private trial.
Required issue labels were provisioned in the otherwise empty repository.
Workflow-run analysis used the default-branch compiled workflow.
Test artifacts were uniquely selected per failed job and attempt.
Same-run replay did not duplicate the occurrence.
Mismatched and injected evidence failed closed.
Issue text was not used as trusted history during migration.
Summary
Scenario
Status
Notes
Focused Infrastructure.Tests
✅ Passed
427/427
gh-aw lock regeneration
✅ Passed
No diff
Multi-test/multi-job shared cause
✅ Passed
One cause issue
Immutable replay
✅ Passed
One occurrence for the same attempt
Legacy migration
✅ Passed
Three rows and operator note preserved
Mismatched test/job evidence
✅ Passed
Rejected before publication
Untrusted test evidence
✅ Passed
Rejected before publication
Overall Result
✅ PR VERIFIED
PR #20333's final workflow grouped multiple verified failures from different
jobs into one cause issue, preserved immutable and legacy occurrence history,
migrated the issue safely, and rejected invalid evidence without mutating
published state.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
the
area-engineering-systems
Summary
CI failure tracking treated the test that happened to fail as the durable
identity of the failure. That breaks when one mechanism affects several tests,
or later appears under a different test or job.
Tests and jobs now describe where a failure was observed. The underlying
mechanism identifies the long-lived cause and issue.
What broke
#19455 correctly made CI attribution and publication fail closed, but left one
scalar
test_nameserving as both cause identity and display text.That model could not represent two valid cases:
When analysis grouped several browser-debugger timeouts, publication looked for
one test with the combined display name and stopped:
The run received neither its expected issue occurrence nor its analysis
comment. A later recurrence under another test could also create a duplicate
cause issue instead of extending the existing one.
Why this model
The publisher therefore verifies trusted evidence, unique ownership, and stored
identity. It does not guess whether two similar failures share a root cause.
Legacy data
Existing cause history is preserved rather than rewritten into the new
shape.
operator-authored notes remain intact.
replay evidence identifies the attempt. Known attempts use exact
/attempts/Nlinks, including attempt 1, so later retries cannot change whata historical row points to.
test_nameand job labels remain descriptive evidence. They arenot promoted into new verified identity or assigned invented numeric job IDs.
mainscope only from the matchingtrusted run summary. Otherwise missing context renders as
unavailable;editable issue text is not used to guess it.
mainscope wins over stale legacy PR metadata. Invalid cachedcontexts such as
#0are rebuilt rather than preserved.new display fields, rows are derived deterministically from stored legacy
evidence, never from editable issue text or fresh analyzer output.
human-authored content.
This keeps existing tracking issues useful while allowing future occurrences to
use the stronger evidence model.
Issue history and recovery
The issue body is a projection, not a second occurrence database. Rebuilding it
from persisted history means retries can repair stale, edited, missing,
duplicate, or previously trimmed rows without changing the underlying evidence.
Text outside the managed occurrence section remains operator-owned. Pull-request
code failures also remain PR-comment-only; this change does not turn them into
persistent cause issues.
Regression coverage
The 427 passing workflow tests cover:
recreation from stored history;
operator-note preservation, and deterministic fallback for malformed legacy
context.
The generated workflow was regenerated with the repository-pinned gh-aw v0.88.7
compiler. No action pin changed.
Fixes #20332
Checklist
<remarks />and<code />elements on your triple slash comments?