Intermittent CodeQL `Analyze (go)` Autobuild runner termination blocks merges, and GitHub refuses to re-run it

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

Measured 2026-08-29 03:36Z across every open PR, with a main control:

PR	Analyze (go)
#6742	success
#6764	failure
#6765	failure
#6766	success
main, 5 newest settled runs	success ×5

So it is intermittent, not systemic — same code, both outcomes. The failing job's annotations are
cgo extraction errors, not analysis findings:

could not import C (no metadata for C)
cannot convert w.window (variable of Pointer type pointer) to type windowPointer

Why this blocks merges

main carries a code_scanning ruleset rule (CodeQL: alerts=all, security=all) in addition to
required_status_checks (CI - Required Checks). A run that fails to produce Go results therefore
gates the merge even though Analyze (go) is not in the required-status-checks list. That
distinction is easy to miss: reading only required_status_checks suggests CodeQL is advisory, and it
is not.

Both affected PRs sit BLOCKED with every other check green — including #6764, a release PR with
auto-merge already armed, so this stalls the release train rather than one feature.

Why it does not clear itself

The analysis is GitHub-managed (event: dynamic, path: dynamic/github-code-scanning/codeql), so
there is no workflow file in the repository and gh run rerun is refused outright:

This workflow run cannot be retried

The only way to get a fresh analysis is a new head commit. For a bot-authored release PR nothing
will produce one, so it can sit blocked indefinitely.

Expected

An intermittent extractor failure should not permanently block a merge, and recovering from one should
not require inventing a commit.

Acceptance criteria

• Root-cause the cgo extraction failure (which package pulls cgo into the Go analysis, and whether
  it can be excluded from CodeQL's build without losing coverage of first-party code).
• Either stabilise the analysis, or move off default setup to an advanced-setup workflow so the run
  lives in this repository and can be re-run.
• A blocked PR has a documented recovery path that does not depend on an empty commit.
• Confirm the intended code_scanning rule behaviour is what we want for an extractor error
  (as opposed to a real alert).

Size

Medium. The diagnosis is bounded; the remedy is likely the default-setup → advanced-setup move, which
is a known, reviewable change.

Blocker: ksail repository settings — CodeQL default setup must be disabled, and the "Require code quality results" ruleset needs a source once it is | authority | last-verified 2026-09-20: default setup still configured (actions, go, javascript, javascript-typescript, typescript; suite extended), and rulesets Require code scanning results (6455804) and Require code quality results (16885082) are both active. The fix (#7130) is written and its own contract failures are resolved, but every Analyze job fails CodeQL analyses from advanced configurations cannot be processed when the default setup is enabled until the setting is changed, so the workflow cannot be validated. | asked pr 2026-09-19

[devantler]

🤖 Generated by the Agentic Engineer

Correction: the cgo diagnostics are not the cause — the runner is being killed

The blocking mechanism above is right and unchanged. The root cause is misattributed, and
acceptance criterion 1 as written sends the fix down a dead end.

The control

The could not import C / cannot convert ... windowPointer lines are present in successful runs
too, so they cannot be what distinguishes a failure:

Run	Head	Conclusion	could not import C	received a shutdown signal
33230338250 (main)	a8cff518	success	3	0
33229392451 (#6765)	a4503b1e	failure	3	1
33225677735 (#6764)	6f3e88e2	failure	3	1

CodeQL tolerates partial cgo extraction — that is ordinary noise in this repository's Go analysis.
The one line that appears in both failures and in neither success is:

##[error]The runner has received a shutdown signal. This can happen when the runner service is
stopped, or a manually started runner is canceled.

Both failing jobs die in the Autobuild step (#6764 at 01:21:53Z, ~13 min in; #6765 at 02:55:54Z,
~9 min in). That is an infrastructure kill, not an extractor defect.

Frequency

4 failures in the 200 most recent runs of dynamic/github-code-scanning/codeql (~2%), all inside ~24h:
#6765, #6764, #6760, and one on main (2026-08-28T08:10Z).

Recovery: close/reopen does NOT work

Worth recording as a negative result, because it is the obvious first thing to try. I closed and
reopened #6764 (head unchanged at 6f3e88e2): zero new runs were created at that head afterwards.
Default-setup code scanning does not re-analyze on a reopened event, so the "new head commit" finding
above is stronger than stated — it appears to be the only recovery. rerun-failed-jobs remains a hard
403 This workflow run cannot be retried.

Suggested revision to the acceptance criteria

• Root-cause the Autobuild runner termination — the leading hypothesis is resource exhaustion
  while autobuilding a heavy cgo dependency tree (the wails/v3 graph), which would explain why it
  is intermittent and why it always lands in Autobuild. Confirm before acting on it.
• Excluding cgo packages from the analysis is not on the critical path — the cgo diagnostics are
  present in passing runs. Do not spend effort there on this evidence.
• Moving to advanced setup remains the strongest fix, and now for a second reason: it makes the run
  re-runnable, which is the missing recovery path.

The rest of the issue stands as written.

[devantler]

🤖 Generated by the Agentic Engineer

Strengthening the close/reopen result — the event did fire

My earlier note said close/reopen produced no new analysis. That was measured against the wrong cutoff,
so here is the proper control. Reopening #6764 (head unchanged at 6f3e88e2) did fire a
pull_request event, and five repository workflows re-ran on it:

Created	Event	Workflow
04:17:12Z	pull_request	enable-auto-merge.yaml
04:17:13Z	pull_request	ci.yaml
04:17:13Z	pull_request	dependency-review.yaml
04:17:13Z	pull_request	scan-for-workflow-vulnerabilities.yaml
04:17:13Z	pull_request	validate-go-project.yaml

Meanwhile CodeQL has exactly one run at that head for the whole PR lifetime — the original
33225677735 from 01:10:26Z, failure. So this is not "the event did not reach the repository": the
event fired, every workflow-file-backed check re-ran, and default-setup code scanning alone did not
re-analyze. The conclusion holds and is now controlled: a new head commit is the only recovery.

Side effect worth knowing: the reopen also re-ran enable-auto-merge.yaml, which re-armed auto-merge
at 04:17:45Z. So close/reopen is safe with respect to the auto-merge state on this repository — it is
simply ineffective for the thing that is actually blocking.

Also confirming the gate experimentally: once CI - Required Checks went green at that head, #6764 was
still BLOCKED with Analyze (go) the only red — so the code_scanning ruleset is doing the
blocking, exactly as this issue states, and not the status check.

[devantler]

🤖 Generated by the Agentic Engineer

Tested a cheaper remedy than a new head commit — it does not work

This issue says "The only way to get a fresh analysis is a new head commit." That is the
expensive claim in here, because a new head stales the green review and costs another metered review
round. There is an obvious candidate that would avoid it — CodeQL default setup runs on
pull_request, and reopened is one of that event's default activity types, so closing and
reopening ought to re-dispatch the analysis at an unchanged head.

I tried it on ksail#6847 and it does not. Recording the negative result so nobody spends the
attempt again.

Action	gh pr close 6847 → gh pr reopen 6847 at 01:07Z, head unchanged at c7f9d393cbb8
Repo's own pull_request workflows	✅ all six re-dispatched at 01:08Z (CI - KSail, Validate Go Project, Dependency Review, Desktop App, Scan for Workflow Vulnerabilities, Enable Auto-Merge)
GitHub-managed CodeQL run at that head	❌ none — still exactly one, 33689878970 from 22:20:24Z
CodeQL merge gate	❌ unchanged: neutral — 1 configuration not found, started_at still 22:21:11Z
PR state	still BLOCKED

So the reopen reaches repository-defined workflows but not the managed code-scanning
configuration. Both re-run endpoints stay refused as documented — rerun and rerun-failed-jobs each
return 403 This workflow run cannot be retried.

One side effect worth knowing, because it is misleading: after the reopen the Analyze (go),
Analyze (javascript-typescript) and Analyze (actions) check-runs disappear from the head
entirely. The head then carries no failing check at all — only the neutral gate. That is a trap
for the merge preflight's documented staleness exception, which applies when every check-run is
success/skipped: neutral is neither, so the exception correctly does not apply, but a preflight
that skims for "any failures?" will now see none and read the block as stale.

Correction to the main control in the issue body

The body records main, 5 newest settled runs: success ×5, and concludes the failure is not
base-inherited. Over a wider window that does not hold — 3 of main's 8 most recent CodeQL runs
failed (33671294945, 33668578090, 33660820603).

I pulled the log of one of them as a positive control. Run 33671294945, commit c53ef329 on
main, unrelated to any of the PRs here, carries the identical signature:

##[error]... /go/pkg/mod/github.com/wailsapp/wails/v3@v3.0.0-alpha.98/internal/assetserver/webview/request_linux.go:11:8: could not import C (no metadata for C)

So the extractor failure is not PR-specific and not intermittent-by-luck alone — the base branch
reproduces it at roughly the same rate. The 5-run sample simply missed it. That strengthens the case
for the root-cause fix rather than a retry mechanism: buildless Go analysis (build-mode: none under
advanced setup) removes autobuild, the ~19-minute extraction, and the cgo import errors together.

Current impact

This is not hypothetical right now. main is red on CI - KSail → 🛡️ Vulnerability Scan
(Reachable vulnerabilities not accepted in ./.govulncheck-allow.txt: GO-2026-6354 GO-2026-6355), and
ksail#6847 — the fix for exactly those two advisories, with a green Codex review at head, zero
review threads, and CI - Required Checks green — is held out of main solely by this gate.

[devantler]

🤖 Generated by the Agentic Engineer

The cgo errors are a red herring — both observed failures end in a runner shutdown

Re-diagnosed 2026-09-03 by reading the job logs rather than the check-run annotations. The
annotations surface the cgo extraction errors, so those look like the cause; they are not.

Both failures I could read terminate with the same final line, and it is not a cgo error:

##[error]The runner has received a shutdown signal. This can happen when the runner service is
stopped, or a manually started runner is canceled.

head	outcome	Analyze (go) duration	terminal log line
6947df1545 (#6849)	failure	28 min	runner shutdown signal
c53ef3292 (main)	failure	14 min	runner shutdown signal
71f35b7701 (main)	success	20 min	—
fc85581425 (main)	success	24 min	—
8b2582c220 (main)	success	22 min	—

Why the cgo errors are not the cause

They are non-fatal. In the same log, extraction continues past them — CodeQL logs
Skipping dependency package … for the affected packages and then completes a further extraction
pass. The failing packages are all Wails v3 Linux bindings pulled in by the desktop/ module
(request_linux.go, webkit_linux.go, application_linux.go, linux_cgo.go), i.e. missing
GTK/WebKit dev headers on the runner. They are noise on every run, including the successful ones.

Duration does not predict it either

That was my first hypothesis and the data refutes it: a 14-minute run failed while 20-, 22- and
24-minute runs succeeded. So "the job is too long and gets clipped" is not supported — this reads as
runner preemption independent of how long the job has been going.

⚠️ n = 2 failures. That is enough to say the terminal cause is a shutdown signal in both cases and
enough to refute the cgo and duration stories, but not enough to characterise why the runner is
reclaimed. I have not established that.

What this changes about the acceptance criteria

• AC-1 ("root-cause the cgo extraction failure") rests on a false premise. Worth keeping as
  separate hygiene — the noise is real and makes every log harder to read — but fixing it will not
  stop the blocking, so it should not gate this issue.
• AC-2 (move off default setup to advanced setup) is now the load-bearing remedy, and for a
  sharper reason than before: if the cause is runner preemption, it is not something this repository
  can prevent — so the thing that matters is being able to recover. Under default setup the run is
  GitHub-managed and gh run rerun is refused; under advanced setup the workflow lives here and a
  failed job is re-runnable.
• AC-3 follows from AC-2 — with advanced setup the recovery path is a re-run rather than an
  invented commit.
• AC-4 stays as written and gets sharper: a code_scanning rule with alerts=all currently
  cannot distinguish "the analysis found nothing" from "the analysis was killed by infrastructure".

Live instance right now

#6849 (chore(copilot-plugin): release v7.181.8, ksail-bot, auto-merge armed by its own
automation at 03:02:55Z) is BLOCKED on exactly this: every other check green, CodeQL neutral with
"1 configuration not found — /language:go". Auto-merge is armed but can never fire, so the release
train is stalled again, which is the second time this issue has caught a release PR.

Current default setup for reference: languages actions, go, javascript, javascript-typescript, typescript, query suite extended, threat model remote, runner_type: standard.

[devantler]

🤖 Generated by the Agentic Engineer

New live instance — this now blocks a release PR

#6849 (chore(copilot-plugin): release v7.181.8) is hard-blocked by this exact
failure mode, which raises the impact beyond dependency bumps: it is now blocking the
release path.

Evidence, run 33709911261, job Analyze (go) (03:02:58Z → 03:31:08Z):

• The job ends at ##[error]The runner has received a shutdown signal. — preemption,
  not a code defect.
• The cgo could not import C lines earlier in the same log are not the cause. They
  are build-stderr noise that green runs on main carry too (control recorded on
  ci(codeql): replace managed default setup with re-runnable advanced setup #6850's investigation).
• rerunnable and rerunnable_failed_jobs are both null, and the run is
  event: dynamic. gh run rerun returns 403 — "This workflow run cannot be
  retried", and that holds even though the run is completed, not only while it is in
  progress.

So there is no re-dispatch path at all for this head: a new commit is the only way to
get another CodeQL run, and a ksail-bot release PR does not produce one on its own.
Require code scanning results is an active ruleset, so the PR sits BLOCKED with
auto-merge armed and nothing that can ever satisfy it.

This is the same mechanism as the main streak measured on 2026-09-02, where two
consecutive main runs failed (18:39:43Z, 19:06:25Z) and only self-healed at 19:24:10Z
because main gets a later run. A PR head has no such next run.

Bearing on #6850: the advanced-setup migration is what makes these runs re-runnable,
so it is not only a CI-hygiene improvement — it is the unblock for stranded release and
dependency PRs. #6850 remains parked on the maintainer disabling CodeQL default setup
(Settings → Code security → Code scanning → Default setup); until that flip, its own
Analyze (*) legs run duplicated alongside default setup.

[devantler]

🤖 Generated by the Agentic Engineer

Reframing: the advanced-setup remedy is ruled out — remedies must keep default setup enabled

#6850 attempted the obvious fix (replace managed default setup with a re-runnable advanced setup) and
I have closed it, because the evidence says the approach cannot work here and the step that would
unblock it is actively harmful. Re-verified live today at 6b1e1da1:

• Advanced setup cannot run at all while default setup is on —
  CodeQL analyses from advanced configurations cannot be processed when the default setup is enabled
  — and code-scanning/default-setup is state: configured. The PR can never demonstrate itself
  before the flip it is meant to justify.
• After a flip, the active Require code quality results ruleset (16885082, severity: all)
  loses its producer: this repo's own run log shows analysis-kinds: code-scanning,code-quality
  degrading to code-scanning only, with the input documented as "experimental and for
  GitHub-internal use only". Every PR would then block on a result nothing emits, or the quality gate
  would silently stop gating.

So the constraint for this issue is now: any remedy must keep default setup enabled.

One thing I checked that turns out not to be a lever

Require code scanning results (6455804, on ~DEFAULT_BRANCH) is configured
alerts_threshold: all, security_alerts_threshold: all. Those thresholds decide which alert
severities block — they do nothing about a missing or errored analysis, which is this issue's
actual failure. Loosening them would weaken the gate without unblocking anything, so it is not a
candidate.

Candidate directions, with what is known and what is untested

A. Fix the intermittency at its source (root cause). The failing annotations are cgo extraction
errors, not findings:

could not import C (no metadata for C)
cannot convert w.window (variable of Pointer type pointer) to type windowPointer

Same commit succeeds and fails across runs, so the extractor is tripping over cgo non-deterministically.
This is the only direction that removes the problem rather than routing around it. Untested: whether
build tags, or isolating the cgo package, makes extraction deterministic.

B. Re-trigger a managed analysis without a new commit. Managed runs are event: dynamic and
gh run rerun is refused, but updating the default-setup configuration is documented to trigger a
fresh scan. Untested here, and the open question is whether that re-analyses open PR heads or
only the default branch — if only the latter, it does not help a stuck PR. Cheap to measure, and worth
measuring first because it would be a general unblock.

C. Produce a new head commit for the stuck class. The permanent-stick case is narrow: a
bot-authored release PR that will never push again. A trivial real commit clears it. This is a
workaround, not a fix, and it should be scoped to that class rather than made a general habit.

Suggested order

Measure B first (cheapest, and a general unblock if it works), then invest in A as the durable
fix. Keep C as the escape hatch for a release PR that is already stuck.

Note on how often this actually bites

Managed CodeQL here is intermittent, not a streak — main failed 2026-09-02 at 18:39Z and 19:06Z
and recovered at 19:24Z without intervention. A PR only sticks permanently when nothing will push to
it again, which is why the release-PR case is the one that matters.