dotnet.yml's build job wedged and was cancelled at its cap — and the logs were discarded, so there is no evidence

[jeremydmiller]

The build job in .github/workflows/dotnet.yml wedged once on 2026-08-24, ran to its cap, and was
cancelled — and because of how a cancelled job is handled, every piece of evidence about it is
gone. This is the same class of failure as #4083, in the one workflow that has no instrumentation.

Filing it because the occurrence is real and the next one will be just as unreadable.

What happened

Attempt 1 of run 32737221654, on
PR #4090:

job	build (97462820888)
conclusion	cancelled — not failure, and no step marked failed
ran	14:48:43 → 15:13:43 = 25m00s
Test step	started 14:49:01, never completed
logs	gone — BlobNotFound

The 25m00s is worth a note: the job's cap is timeout-minutes: 20, and GitHub allows a job up to a
further 5 minutes to wind down after cancellation before force-terminating it. 20 + 5 = exactly the
25m00s observed, which suggests the process did not merely overrun the cap — it also failed to die
when asked, using the entire grace window. That is a wedge, not a slow run.

It is far outside normal

Across the last 25 green runs of this workflow, the build job takes:

n=25   min 458s   median 616s   max 738s
slowest five: 700, 706, 709, 720, 738

So the norm is ~10 minutes and the worst on record is 12m18s, against a 20 minute cap. The wedged
attempt was still going at 24m42s in the Test step alone — more than double the slowest healthy
run, and it never produced a result.

Not caused by the PR it appeared on: #4090's diff is .github/workflows/tests.yml and
build/ci-memory-sampler.sh, and this workflow runs ./build.sh ci, which references neither. A
re-run went green and #4090 merged on 38/38.

Why this is worth an issue rather than a shrug

It is rare — 0 cancelled runs in the last 100 — but the current setup loses the evidence twice
over:

1. Cancellation discards the logs. Not truncates: BlobNotFound. Nothing survives except the
   per-step timestamps in the API.
2. Re-running overwrites the run's conclusion. That occurrence now reports success, because I
   re-ran it. Scanning history for conclusion == "cancelled" finds nothing — the only reason this
   report exists is that someone happened to be looking at the time.

And, exactly as in #4083, gh pr checks renders a cancelled job as fail, which reads as a test
failure and sends the next person hunting for a broken test that does not exist. Confirm the real
state with:

gh api repos/JasperFx/wolverine/actions/jobs/<id> --jq '"\(.status) \(.conclusion)"'

Suggested fix

dotnet.yml is now the only test-running workflow with no memory sampler. build/ci-memory-sampler.sh
(#4084, widened to all 34 targets in #4090) already does exactly what is needed here, and it exists
precisely because a cancelled job's later steps never run and its log is not retained — it streams to
the live log as it goes, and on a stall it captures the wedged process's async stacks with
dumpasync before the cap can take them.

Wiring it into this workflow is a handful of lines, mirroring the Run Tests step in tests.yml:

      - name: Test
        env:
          CI_JOB_NAME: dotnet-ci
        run: |
          ./build/ci-memory-sampler.sh &
          sampler_pid=$!
          trap 'kill "${sampler_pid}" 2>/dev/null || true' EXIT

          ./build.sh ci

The sampler must be backgrounded from that shell specifically — it picks the process to watch out of
its own parent's process tree, which is how it distinguishes the test host from a service container.

The default 780s deadline is a reasonable fit here: this job's measured green max is 738s, which is
close, so the deadline may want to be 0 (idle detector only) as it is for the heavy targets in
tests.yml. Worth deciding with a few more data points rather than guessing.

Refs #4083. Refs #4084.