Skip to content

ci: Consolidate test results reads the PREVIOUS attempt's shard artefact after rerun-failed-jobs (latest-by-id is the older upload) — a shard that re-ran green cannot turn the required check green #4303

Closed

Description

@rbuergi

Consolidate test results consolidates the PREVIOUS attempt's shard artefact after rerun-failed-jobs — a shard that re-ran green cannot turn the required check green

Measured, run 34823051999 (#4292, head 75ae0835)

attempt Run tests (shard 1) Consolidate test results why
1 ❌ one failing test (ReadPathStreamMintingTest.AnUncachedConfiguredReduce_MintsOneSyncHubPerCall, #4300) ❌ correct
2 (rerun-failed-jobs, 08:46Z) ✅ 1 454/1 454, artefact testResults-shard1 id 10338734845 uploaded 08:49:55Z ❌ ##[error]MeshWeaver.Data.Test.ReadPathStreamMintingTest.AnUncachedConfiguredReduce_MintsOneSyncHubPerCall · [CI] MeshWeaver.Data.Test exit=1 TESTFAIL it downloaded testResults-shard1 (ID: 10338854604, Size: 297521) — attempt 1's artefact (uploaded 08:38:28Z)

The run now holds two artefacts named testResults-shard1:

id 10338854604  size 297521  created 08:38:28Z   ← attempt 1, carries the failure
id 10338734845  size 298484  created 08:49:55Z   ← attempt 2, green

actions/download-artifact@v8 with pattern: testResults-shard* reports Found 12 artifact(s) — 6 build-output-* + 6 testResults-shard* after its own per-name de-duplication — and the one it keeps per name is the highest id, which here is the OLDER upload: artefact ids are not monotonic across attempts (attempt 2's id is lower). So the consolidate reads the stale trx and re-declares the failure the re-run just cleared. The exit-marker gate reads the same directory and agrees, for the same reason.

Falsified as predicted — attempt 3 (rerun-failed-jobs of the consolidate alone, 08:52:50Z, job 103914885830): Found 12 artifact(s) → testResults-shard1 (ID: 10338854604, Size: 297521) again → the same ##[error] on the same test at 08:53:09Z, 3 min after the listing was provably settled (both artefacts visible over REST at 08:52:50Z). The choice is by id and structural, not a listing lag. This run cannot be made green by any re-run; only a new head (a new run id with fresh artefacts) can.

Why this matters

Fix direction (not band-aids)

The artefact a shard uploads should be attributable to its attempt, and the consolidate should read only the current attempt's set — e.g. name shard artefacts with ${{ github.run_attempt }} and filter on it, or download each shard's artefact by the ID the shard job outputs, never by name pattern. Deleting the stale artefact after a re-run is the same fix from the wrong end.

Refs #4300 (the unattributed failure that exposed it), #4127 (a re-run that passed by luck of the id), #2921.

Activity

  1. added a commit that references this issue on Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions