ci: stop waking the shared GB10 runner for changes that cannot affect what it builds #1992

Context

The changes job's path filters in .github/workflows/ci.yml:122-191 over-trigger in three distinct ways, and all three land on GB10, the single self-hosted Linux runner that also serves the release build. Six jobs target it: cargo-clippy (ci.yml:228-241), WebUI installed artifact (ci.yml:566-570), OpenXLA feature compile (ci.yml:941-945), OpenXLA feature link (ci.yml:1112-1116), CUDA sm_70 compile (ci.yml:1203-1207), and CUDA block-float converter (ci.yml:1359-1363). The workflow already documents the contention this causes in its concurrency block (ci.yml:64-95): "Measured over 2026-09-10 KST: 113 runs created, 14 open at once, 11 of those stacked on four branches, and six cancelled by hand purely to free the runner (#1774)."

Defect 1: every filter includes the workflow file itself. All six filters list '.github/workflows/ci.yml': rust at ci.yml:129, mlx_pin at :136, xla_link at :146, cuda_arch at :163, rocm at :177, webui_bundle at :191. Editing one job's comment therefore re-runs all six GB10 jobs. Demonstrated on PR #1989, whose entire diff was .github/workflows/ci.yml and docs/installation.md with zero Rust or C++ changes: all six GB10 jobs were triggered, GB10 was down, the run sat queued about six hours and never completed. Because the run never completed, gh run view --log returned "run is still in progress; logs will be available when it is complete" for every job in it, so no job's output was readable and the merge needed a bypass of pending checks. This defect was already noted in passing during #1774 (TECHNICAL_REPORTS/1774-ci-concurrency-group-20260911.en.md:74) and not acted on.

Defect 2: the CUDA architecture filter covers the ROCm and Metal overlays. cuda_arch lists 'src/lib/mlx-cpp/**' (ci.yml:158-159), and that tree contains patches/ (Metal, 28 files), patches-cuda/ (3), patches-rocm/ (126) and turbo/ (24). A change confined to patches-rocm/** therefore wakes CUDA sm_70 compile and CUDA block-float converter on GB10, two jobs a HIP source file cannot affect. Most of epic #1801's PRs are exactly this shape.

Defect 3: the ROCm filter has the mirror problem. The rocm filter added in #1989 also lists 'src/lib/mlx-cpp/**' (ci.yml:169-170), so a Metal-only or CUDA-only overlay change trips the ROCm gate. Same defect, opposite direction, introduced by the same pattern.

The reason ci.yml sits in every filter is real and documented: ".github/workflows/ci.yml is included so edits to this job itself are testable" (ci.yml:1043-1044), guarding the failure class where a missing path trigger let the IREE link regression of #1274/#1275 reach main (ci.yml:915-919, ci.yml:1005-1013). But that protection is aimed at the wrong half of the problem. CI-change failures come in two classes. Schema and syntax errors, where GitHub rejects the whole file and runs nothing, so re-running heavy jobs catches nothing and the run just fails with "This run likely failed because of a workflow file issue"; that happened during #1989, caused by permissions: administration: read, which is not a valid GITHUB_TOKEN permission key, as the surviving comment at ci.yml:1729-1734 records, and a YAML parse accepted it where a hosted linter would have rejected it in seconds. And behavioral errors in a job's own script, where only running that job helps. The current design pays the full six-job GB10 cost on every workflow edit, covers the second class for jobs nobody touched, and covers the first class not at all.

Scope

In scope: .github/workflows/ci.yml only, specifically the changes filter definitions (:122-191), each heavy job's if expression, and two new hosted jobs. Plus one new repository label, ci:full.

Out of scope: release.yml, nightly-verify.yml, pipeline-parallel-ci.yml, python.yml, update_homebrew_formula.yml. Splitting ci.yml into per-job workflow files (see Rejected alternatives). Changing what any heavy job runs; only when it runs changes.

Implementation plan

1. Add a hosted workflow-lint job triggered on any .github/workflows/** change, running actionlint (it validates the permissions key set, expression syntax, and shellcheck findings inside run: blocks). This is new coverage for the class that actually broke a run, costs seconds, and does not touch GB10. Baseline it: actionlint 1.7.7 reported 8 pre-existing findings on this file (unknown self-hosted runner labels, shellcheck info notices) per TECHNICAL_REPORTS/1774-ci-concurrency-group-20260911.en.md:70, so pin the version and either suppress those categories or accept the current count, otherwise the job lands red.
2. Remove '.github/workflows/ci.yml' from the rust, mlx_pin, xla_link, cuda_arch, webui_bundle and rocm filters (ci.yml:129, :136, :146, :163, :177, :191), and update the filter and job comments that justify its presence, including ci.yml:1043-1044.
3. Add an opt-in ci:full PR label that ORs into each of the six heavy jobs' if expressions, so a change to a heavy job's own script can still be validated deliberately before merge. The label does not exist in this repository yet and must be created with the intended color and description.
4. Add an advisory hosted job: when the diff touches .github/workflows/** and the ci:full label is absent, emit a ::warning:: naming the self-hosted jobs whose definitions changed without being re-run. Advisory only, never failing, because blocking would reintroduce the cost it exists to report. ci(rocm): build, link and generate on a self-hosted gfx1151 runner #1989 already established this pattern with ROCm CI status (ci.yml:1705-1741, warning emitted at :1739); follow it.
5. Narrow cuda_arch to exclude the non-CUDA overlays using negated patterns ('!src/lib/mlx-cpp/patches-rocm/**', '!src/lib/mlx-cpp/patches/**'), and narrow rocm symmetrically to exclude patches/** and patches-cuda/**. Keep turbo/, CMakeLists.txt, the build scripts and the MLX pin paths in both, since those are genuinely shared. No filter in this repository currently uses a negated pattern, so confirm dorny/paths-filter@v4 negation semantics against its documentation before relying on them, and state in each filter comment why every exclusion is safe.

Rejected alternatives

Removing ci.yml from the filters with no replacement: loses the ability to validate a CI change before merge, which is the gap that let #1274 through, so steps 1, 3 and 4 exist to replace that protection rather than drop it.

Splitting each heavy job into its own workflow file with its own on.pull_request.paths: the most precise option, since editing one file would trigger only its own jobs, but it is a large refactor of a 1741-line file carrying dense rationale comments and a shared changes job, so it belongs in a separate change if the above proves insufficient.

Parsing ci.yml to map changed line ranges to job names: more precise still, but fragile and unpredictable for a reviewer trying to anticipate what will run.

Acceptance criteria

• A PR whose diff is only documentation and workflow comments triggers no GB10 job.
• A PR touching only src/lib/mlx-cpp/patches-rocm/** triggers the ROCm gate and neither CUDA GB10 job.
• A PR touching only src/lib/mlx-cpp/patches/** or src/lib/mlx-cpp/patches-cuda/** does not trigger the ROCm gate.
• An invalid workflow file, for example a bad permissions key, fails the hosted lint job with a message naming the key.
• A PR carrying ci:full runs the six heavy jobs regardless of paths.
• A PR changing a heavy job's script without ci:full gets an advisory warning naming that job and is not blocked.
• The ci:full label exists in lablup/mlxcel with a description explaining when to apply it.
• The filter comments in ci.yml state why each exclusion and each removal is safe, so the next reader does not restore them.
• Integrated into the real workflow file on main, not left as a proposal or a side workflow.

Validation

Verify filter behavior without burning runner time, then confirm once live:

# 1. Lint the edited file locally, exactly as the new hosted job will.
#    actionlint is not installed on this host; install it first.
go install github.com/rhysd/actionlint/cmd/actionlint@v1.7.7   # or download the release binary
actionlint .github/workflows/ci.yml

# 2. Negative control: the linter must reject an invalid permissions key rather
#    than accept it as valid YAML (the #1989 failure).
/bin/cp .github/workflows/ci.yml /tmp/ci-bad.yml
# add `administration: read` under a job's `permissions:` in /tmp/ci-bad.yml
actionlint /tmp/ci-bad.yml   # must fail naming the key

# 3. Check the filter patterns against dorny/paths-filter's documented negation
#    semantics, and dry-run the match set per shape with the same globbing rules.
git diff --name-only origin/main   # the change's own file set

# 4. One scratch PR per shape, confirming which jobs start:
#    docs-only; workflow-comment-only; patches-rocm-only; patches-only;
#    patches-cuda-only; a Rust change; the same Rust change plus `ci:full`.
gh pr view <n> --json statusCheckRollup -q '[.statusCheckRollup[].name]'

A pass is: step 1 reports no new findings beyond the pinned baseline, step 2 fails, and in step 4 each shape's check list contains exactly the jobs the acceptance criteria name and no GB10 job the criteria exclude. Note that pkg-config and pyyaml are absent from at least one shell on the ROCm host, so a YAML-only parse check there is not a substitute for actionlint.

References

.github/workflows/ci.yml: the concurrency block and its measured GB10 contention note (:64-95), the changes job's filter definitions (:122-191), each GB10 job's if (:240, :569, :944, :1115, :1206, :1362), the "so edits to this job itself are testable" rationale (:1043-1044), and the ROCm CI status advisory job (:1705-1741).

PR #1989 (the demonstration, and the ROCm CI status advisory pattern), #1774 (the GB10 contention that motivated the concurrency group), #1274 and #1275 (the link regression a path-trigger gap let through), epic #1801 (whose PRs are the main victims of defect 2), #1811 (the ROCm runner the rocm filter gates).

TECHNICAL_REPORTS/1774-ci-concurrency-group-20260911.en.md:70 (the actionlint 1.7.7 baseline) and :74 (the earliest record of defect 1).