[automated] Analyze CI Failure opens duplicates when prior-cause context is truncated

[radical]

[automated] This report was produced from a read-only investigation of the CI failure-analysis workflow.

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Analyze CI Failure can create a new ci-failure-cause issue for a flaky test that already has an open canonical issue.

This occurred when Analyze CI Failure run 31204932433 processed CI run 31200870870. The two resulting issues identify the same fully qualified test:

Aspire.Hosting.RemoteHost.Tests.JsonRpcAuthenticationTests.FailedAuthentication_ClosesConnection_AndPreventsFurtherCalls

However, their generated cause IDs differ:

#18840: remotehost-jsonrpc-auth-timeout
#19143: remotehost-jsonrpcauth-rpc-timeout

The workflow loaded 34 prior causes, including the cause linked to #18840. That cause appeared around line 508 of analysis-summary.md, but the agent read only the first 300 lines. It therefore generated a new cause ID.

The publish step reuses issues only through the exact generated cause ID or its exact HTML marker. Because the IDs differed, it could not match open issue #18840 and created #19143.

Expected Behavior

When a flaky-test cause has the same fully qualified test_name as a prior cause, the workflow should reuse the prior cause ID and append the occurrence to its existing issue.

Steps To Reproduce

1. Populate the memory/ci-failure-analysis branch with enough cause files that the matching prior cause is rendered below the portion of analysis-summary.md read by the agent.
2. Trigger Analyze CI Failure for another occurrence of that test.
3. Observe the agent generate a new cause ID because the matching prior cause was not in its read context.
4. Observe the publish step fail the exact-marker lookup and create a new issue instead of updating the canonical issue.

Exceptions (if any)

N/A

Aspire doctor output

N/A

Anything else?

Relevant logic in .github/workflows/analyze-ci-failure.md:

• Lines 393–404 render every prior cause into an unbounded final section of analysis-summary.md.
• Lines 610–670 derive an exact marker from the agent-generated cause ID and search stored/open/closed issues only by that ID or marker.
• Lines 686–707 update an exact match or create a new issue.
• Lines 949–959 instruct the agent to reuse a prior cause when test_name matches.

The relevant matching behavior remains on main; the later August 19 edit only pinned action versions.

The same analysis run also generated #19144 instead of reusing #18720 for the existing 0xC0000142 Windows process-initialization cause, providing a second instance of the same failure mode.

[joperezr]

🤖 Aspire triage pass — 2026-08-21 19:30 UTC

Area: area-engineering-systems
Kind: bug (CI failure-analysis automation)
Repro: partial — conceptual repro in the issue's "Steps To Reproduce" section; requires seeding the memory/ci-failure-analysis branch so the matching prior cause renders past the agent's read window
Affected version(s): N/A — this is repo CI tooling (.github/workflows/analyze-ci-failure.md), not a shipped Aspire package version
Severity: S3 — produces duplicate ci-failure-cause tracker issues instead of updating the canonical one; workaround exists (humans manually dedup/close)

Urgency: ⚪ backlog — no milestone
Reason: internal CI-automation bug, not a product regression or an S1/S2 product bug; does not meet the raised 13.5 bar while it stabilizes

Suggested owner(s): David Pine (@IEvangelist) (backup Adam Ratzman (@adamint))
Evidence:

• David Pine (@IEvangelist) merged Fix docs safe-output base resolution #19446, Fix gh-aw v0.86.2 safe-output base parsing #19445, Upgrade gh-aw compiler to v0.86.2 #19438, Fix PR docs recovery base handling #19236, Make docs target checkout deterministic #19118 in area-engineering-systems (last ~30d)
• Adam Ratzman (@adamint) merged Remove pipeline-scoped Publish-Build-Assets group #19518, Avoid MongoDB credential false positive #19305, Re-enable VS Code extension E2E tests in CI #19142 and closed Update extension publishing to use secure publish as microsoft #18018, Include changelog in CLI npm package #17719 as completed in area-engineering-systems
• No CODEOWNERS entry covers .github/workflows/analyze-ci-failure.md
• Note: the most likely domain owner is the issue author, who is excluded from owner suggestions; routing here is evidence-based and may need reassignment

Confidence: medium
Needs human?: no

[radical]

[automated] Proposed follow-up design based on the investigation and review of the closed #19804 approach:

Canonical cause selection for Analyze CI Failure

Status

Design handoff for a follow-up change.

This is not the current behavior and is not part of PR #20333.

Related:

• Issue [automated] Analyze CI Failure opens duplicates when prior-cause context is truncated #19578: an existing cause was not considered and a duplicate cause/issue was created.
• PR fix(ci): Publish multi-test flaky causes to one issue #20333: fixes occurrence evidence, persistence, replay, migration, and issue publication.
• Closed PR fix(ci): Prevent duplicate recurring CI failure issues #19804: explored deterministic cause resolution, but its resolver design should not be reused.

Outcome we want

The existing Copilot analysis agent should decide whether a current failure:

1. Matches an existing cause and must reuse that exact stored cause ID.
2. Represents a genuinely new mechanism and needs a new cause ID.
3. Is ambiguous and should not be forced into an existing cause.

Deterministic code should guarantee that the agent receives a complete, bounded, and current cause catalog. It should validate the mechanical contract, but it should not attempt to determine semantic equivalence between failures.

Trusted current-run evidence
           ↓
Copilot classifies the failure mechanism
           ↓
Complete catalog of prior causes
           ↓
Copilot chooses:
  ├─ exact existing cause ID
  ├─ new cause ID
  └─ ambiguous / insufficient evidence
           ↓
Deterministic validation
           ↓
Persistence and issue publication

Why this is needed

The current workflow asks Copilot to semantically compare current failures with prior causes. The publisher then treats the ID chosen by Copilot as the persistent identity.

If the relevant prior cause is not considered, Copilot can propose a different ID for an existing mechanism. Exact-ID persistence then creates a second cause and potentially a duplicate issue.

The underlying problem is incomplete or ineffective cause visibility, not a need for fuzzy ID matching.

Terminology

Term	Meaning
Occurrence	One cause observed in one CI run attempt
Execution identity	(run_id, run_attempt)
Job identity	Numeric GitHub Actions job_id
Test observation	An exact test failure in a specific job
Proposed cause	Copilot's hypothesis about the current failure mechanism
Proposed cause ID	A new ID suggested by Copilot when no existing cause matches
Existing cause ID	The exact ID of a stored prior cause selected by Copilot
Cause mechanism	The underlying reason failures occur across runs, jobs, or tests
Cause catalog	Complete bounded index of stored causes available to the agent
Cause details	Richer record loaded only for plausible catalog candidates

Avoid:

• "Fingerprint" for descriptive diagnostics.
• "Test identity" as cause identity.
• "Similar ID" as evidence that two causes are the same.
• "Canonical cause" for every current memory file until known duplicates are consolidated or marked.

Design principles

1. The agent performs semantic mechanism matching.
2. The validator authenticates evidence and enforces structural rules.
3. IDs are exact keys. No fuzzy ID matching is allowed.
4. Test and job identities describe occurrences, not causes.
5. The agent must receive every eligible prior cause.
6. Incomplete cause data must fail closed for new-cause publication.
7. A false merge is more damaging than a duplicate. Ambiguity must not force reuse.
8. Stored historical text is untrusted data and must remain bounded, sanitized, and structurally isolated.

Current limitations

Prior causes appear late

The workflow currently places prior causes after potentially large logs, annotations, test evidence, PR context, and retry-pattern information. All files may be collected, but a matching cause can still receive insufficient model attention.

Current cause metadata is descriptive

Stored error_pattern values are agent-authored descriptions. They are useful semantic evidence for Copilot, but they are not repository-owned executable matching rules.

Existing retry patterns do not identify causes

eng/test-retry-patterns.json classifies failures as retryable. Its patterns do not identify canonical causes and do not contain cause IDs.

Current memory contains known duplicate cause IDs

The memory branch currently contains cause records known to represent the same mechanisms. A complete catalog cannot identify which duplicate should be authoritative without cleanup or explicit supersession metadata.

Known examples include:

remotehost-jsonrpc-auth-timeout
remotehost-jsonrpcauth-rpc-timeout

windows-process-init-failure-0xc0000142
windows-process-init-0xc0000142

Proposed architecture

Use two-stage prior-cause retrieval.

Stage 1: complete compact catalog

Generate a separate catalog before the agent processes large run logs.

Prefer one deterministic JSONL file per cause type:

ci-failure-data/prior-cause-catalog/
├─ flaky-test.jsonl
├─ infra-failure.jsonl
├─ main-repository-breakage.jsonl
└─ manifest.json

Each catalog record should contain only bounded mechanism-selection data:

{
  "id": "browser-debugger-timeout",
  "type": "flaky-test",
  "title": "Browser debugger endpoint timeout",
  "error_pattern": "Browser endpoint did not become ready after 60 seconds",
  "last_seen": "2026-09-20T12:00:00Z"
}

Required fields:

• id
• type
• title
• error_pattern

Optional bounded fields:

• last_seen
• occurrence_count

Do not include in the complete catalog:

• Full occurrence history.
• Issue bodies or comments.
• Large test lists.
• Large job lists.
• Unbounded logs or stack traces.

Catalog manifest

The collector must produce a deterministic manifest:

{
  "memory_commit_sha": "<sha>",
  "source_file_count": 143,
  "catalog_entry_count": 143,
  "catalog_ids": ["..."],
  "complete": true
}

The exact digest or ID-list representation can be decided during implementation, but the following invariants are required:

• Every eligible source file produces exactly one catalog entry.
• IDs are unique.
• Entries are deterministically ordered.
• The source memory commit is recorded.
• Rendering or parsing failure makes the catalog incomplete.
• No source file may be silently skipped.

Stage 2: exact candidate details

After reading the complete relevant-type catalog, Copilot may load the original stored JSON for plausible IDs:

ci-failure-data/prior-causes/<exact-id>.json

Candidate details can include:

• Recent tests and jobs.
• Issue URL.
• Occurrence count and timestamps.
• Existing historical limitations.
• The complete bounded stored error_pattern.

Only exact catalog IDs may be used to retrieve candidate details.

Agent decision contract

For each tracked current failure:

1. Classify the current failure using current trusted phase and diagnostics.
2. Read the complete catalog for that cause type.
3. Identify plausible existing cause IDs.
4. Load details for every plausible candidate.
5. Compare failure category, failed operation or phase, and current diagnostic mechanism.
6. Reuse an exact existing ID only when the mechanism agrees.
7. Propose a new ID when no existing cause is sufficiently supported.
8. Do not choose between unresolved duplicate IDs unless one is explicitly marked authoritative.
9. Do not reuse a cause solely because:
   • The test name matches.
   • The job name matches.
   • The IDs or titles look similar.
   • Both failures contain a generic timeout or error code.

The existing output schema can continue using membership to distinguish the decision:

ID exists in catalog     → existing cause selected
ID does not exist        → new cause proposed

An extra agent-authored assertion such as checked_all_causes: true is not trustworthy and should not be added.

Deterministic validation

The validator can enforce:

• The catalog is complete and structurally valid.
• Every source cause appears exactly once.
• Catalog IDs match source filenames.
• Catalog fields are bounded and sanitized.
• The chosen existing ID occurs exactly in the catalog.
• The chosen existing cause type matches the proposed type.
• A new ID does not exactly duplicate an existing ID.
• Current jobs and test observations remain authenticated.
• New-cause publication does not proceed from an incomplete catalog.
• Publication uses the expected memory commit or follows an explicit stale-memory policy.

The validator cannot enforce:

• That the chosen existing cause is semantically correct.
• That another existing cause would be a better match.
• That a new cause is not a semantic duplicate.
• That Copilot actually considered every catalog record.
• That equal test names imply equal causes.

These limitations must be explicit in documentation and tests.

Ambiguity behavior

When two existing records appear to represent the same mechanism but neither is authoritative:

Do not guess.
Do not silently merge.
Do not create another duplicate ID.

The implementation should provide a bounded failure or review path. The exact user-facing behavior can be selected during implementation, but it must preserve the evidence and alert maintainers.

Existing duplicate migration

Before calling the catalog entries canonical, handle known duplicate records.

Preferred direction:

1. Select one durable cause ID for each known duplicate family.
2. Merge occurrence history without losing run-attempt identity.
3. Preserve the surviving issue relationship.
4. Update or close duplicate tracking issues with explicit references.
5. Remove or mark the superseded memory record.
6. Ensure the catalog exposes only one selectable authoritative ID.

Whether this is a one-time migration or uses explicit superseded_by metadata is an implementation decision. Avoid indefinite alias machinery unless a durable need is demonstrated.

Memory freshness

The catalog manifest records the memory commit SHA used for analysis.

Before publishing a new cause:

1. Re-read the memory branch head.
2. If it changed, do not assume the prior catalog remains complete.
3. Re-run analysis or reject new-cause publication according to the selected workflow policy.

Existing workflow concurrency reduces races but does not replace this check.

Security and prompt safety

Prior-cause records contain historical agent-authored text.

Requirements:

• Render one structured JSON record per physical line.
• Bound every field.
• Strip control characters and workflow commands.
• Redact secrets using the existing sanitization path.
• Exclude issue bodies, comments, and arbitrary Markdown.
• Tell the model that catalog strings are data, not instructions.
• Preserve completeness failures instead of silently dropping malformed records.

Prompt isolation reduces risk but cannot make model-visible text fully inert. Keep the catalog minimal.

Required tests

Catalog completeness

• Every stored cause appears exactly once.
• Catalog entries are sorted deterministically.
• Source count, rendered count, and unique ID count agree.
• Enough preceding summary data to exceed 300 lines does not affect catalog availability.

Fail-closed behavior

• Invalid JSON fails catalog completeness.
• Filename/ID mismatch fails.
• Duplicate IDs fail.
• Unsupported cause type fails.
• A rendering failure cannot be ignored with || true.
• New-cause publication is rejected when the catalog is incomplete.

Security and bounds

• Prompt-injection text remains a JSON string.
• Newlines, Markdown fences, workflow commands, and controls are escaped or sanitized.
• Secrets are redacted.
• Maximum field and total catalog budgets are enforced without silent truncation.

Agent/output contract

• Exact existing ID with matching type is accepted.
• Existing ID with incompatible type is rejected.
• Unknown valid ID is treated as new.
• A declared new cause cannot reuse an exact existing ID if an explicit disposition is introduced.
• Equal test names do not mechanically force cause reuse.
• Similar IDs do not mechanically resolve to one another.

Freshness

• Catalog manifest contains the memory SHA.
• New-cause publication fails or retries when the memory SHA changes.

Duplicate migration

• Known duplicate families expose only one authoritative selectable ID.
• Occurrence history and run attempts are preserved.
• Replaying an occurrence after migration remains idempotent.

Acceptance criteria

The follow-up is complete when:

1. The agent receives a complete relevant-type cause catalog before large run evidence.
2. Every catalog entry can be deterministically accounted for.
3. The agent can retrieve exact details for plausible existing IDs.
4. Exact existing selections and new IDs are mechanically distinguishable.
5. Incomplete or stale catalogs cannot silently create new persistent causes.
6. No code uses fuzzy ID matching or test-name identity.
7. Known duplicate cause families are migrated or explicitly superseded.
8. Documentation clearly separates semantic agent decisions from deterministic validation.
9. Focused tests cover completeness, safety, freshness, ambiguity, and migration.

Scope recommendation

Implement this as a focused follow-up for #19578 rather than expanding PR #20333.

PR #20333 should continue to own:

• Trusted occurrence evidence.
• Multi-test cause representation.
• Persistence and migration of occurrences.
• Replay-safe issue publication.

The follow-up should own:

• Complete prior-cause catalog and manifest.
• Agent candidate-selection contract.
• Candidate detail retrieval.
• Duplicate cause consolidation or supersession.
• Catalog completeness and freshness validation.

Handoff questions for the implementing agent

Resolve these before coding:

1. Should catalogs be type-specific JSONL files or one combined file?
2. What total-size ceiling should fail the workflow rather than truncate?
3. How should ambiguity be represented and surfaced?
4. Should stale memory trigger automatic re-analysis or a hard publication failure?
5. Should known duplicates be physically migrated or marked with temporary supersession metadata?
6. Which existing issues are authoritative for the known duplicate cause families?
7. Does candidate detail retrieval need a dedicated helper, or can the agent read exact prior-cause JSON files directly?