Consolidate per-run CI fan-out into few lanes; collapse main-push bursts; finish dotfiles single-selector migration

[kyle-sexton]

Workstream of melodic-software/github-iac#78. Decision record from the 2026-07-16 capacity incident (228 queued jobs, 2h+ backlog, oldest job queued 11:59Z unreleased at 14:00Z).

Problem

The per-run fan-out shape optimizes for the wrong constraint. It was built to dodge GitHub-hosted per-job 1-minute billing rounding; on the self-hosted fleet the scarce resource is worker slots, and each micro-job pays ~20-30s of container spin-up + JIT registration + checkout for 7-60s of work. Measured today: ~40-50% of fleet throughput consumed by select-runner preflights alone; dotfiles spends 16 selector jobs + 16 lanes per run (37 jobs, legacy Shape B); standards ~30 jobs/run; main-push bursts (4 pushes in 15s from distribution sync) enqueue 4 full runs with zero collapse.

Target shape (research-verified 2026-07-16, official sources)

Per consumer repo:

1. One selector (Shape A, already live in medley/claude-code-plugins) — completes the epic-mandated single preflight; migrate dotfiles off Shape B.
2. Consolidated lanes instead of micro-jobs: a hygiene lane running all lint/hygiene checks as steps in one worker, using continue-on-error: true + id: per check step and a final aggregation step failing on any steps.<id>.outcome == 'failure' (run-everything-fail-at-end; outcome/conclusion semantics per contexts reference). Language/build lanes only where toolchain or resource isolation genuinely differs (e.g. dotnet build/test). standards' fixture matrix folds into a fixtures lane.
3. Parallel steps where I/O-bound (changelog 2026-06-25): background: true/wait/parallel: — runner v2.335.1 (current fleet image) has the engine; max 10 concurrent background steps; a failing background step surfaces at the join; NOT usable inside composite actions. On 2-CPU workers this buys log clarity and overlap for network-bound checks (lychee), not CPU throughput — default to sequential steps.
4. ci-status fan-in unchanged as the single required check (job-level check granularity; a job skipped by if: reports Success for required-check purposes — keep in-workflow conditionals, never workflow-level paths: filters on required workflows).
5. Main-push burst collapse: keep PR groups (cancel-in-progress: true) as-is; give main pushes group: <workflow>-${{ github.ref }} with cancel-in-progress: false — the documented default (queue: single) then cancels the superseded pending run while the in-progress run finishes (concurrency docs). Do NOT use queue: max here (it serializes every push — deploy semantics, not lint semantics).
6. Per-check visibility inside a lane: per-step collapsible logs + $GITHUB_STEP_SUMMARY (1MiB/step, 20 summaries/job). Annotation counts (10/step, 50/job) are community-reported only — do not design around them.

Sizing interplay

Consolidated lanes get a larger default worker (2 CPU / 4GiB — CPU parity with hosted private-repo runners at 2 vCPU/8GB, RAM bounded by the per-host VM budget; provisioning#133/#134). Fewer, beefier workers replace many 2GiB micro-workers; setup-* toolchain installs amortize once per lane instead of once per micro-job, with actions/cache for dependencies (image already ships zstd for hosted cache parity).

Sequencing

Reusable workflow/template changes here → standards runner-policy/component sync as needed → consumer PRs (dotfiles first: Shape A migration + consolidation together; standards fixtures lane; medley last — it already has detect-changes and the most lanes). Selector contract and approvedSelectorReferences governance unchanged.

[kyle-sexton]

Research directives for the consolidation work here — from the placement-lane session; ties to the melodic-software/github-iac#78 decision record.

• Consolidation primitive must be composite actions, not reusable workflows. A called reusable workflow is a separate job paying full runner spin-up (+ hosted minute rounding where billed) — https://docs.github.com/en/actions/concepts/workflows-and-actions/avoiding-duplication. A composite action runs as a step in the caller's job. Caveat: composites take no secrets: — re-plumb as inputs/env.
• Nesting headroom is 10 levels / 50 workflows per run since 2025-11-06 (https://github.blog/changelog/2025-11-06-new-releases-for-github-actions-november-2025/), not the old 4/20.
• Each matrix cell is a separate job — collapse trivial matrices into in-job loops.
• Parallel-steps (background/wait, 2026-06-25 changelog) could parallelize inside one job — plan/runner-version eligibility unverified; test on the fleet before relying.
• Empirical floor: real per-job overhead on the fleet is ~3.4s (not 20-30s; that figure was observation lag + queue wait). Consolidation economics: ~3.4s × eliminated job + queue-slot relief + hosted rounding where billed.
• Size the setup-overhead win from runner-real timestamps before asserting it. Use the REST jobs endpoint (started_at/completed_at per job/step) or runner logs — never jobs.json observation-indexed times (inflate 3-8x). Post-melodic-software/ci-runner#49, consolidation's durable case is hosted minute-rounding cost + both-worlds consistency + starvation prevention; its throughput relief shrinks once capacity supply is fixed.

— placement-lane session

🤖 Generated with Claude Code

[kyle-sexton]

Concurrency drift findings from the standards#151 audit that belong to this lane (workflow-structure fixes in medley are yours per the lane split):

• medley/ci-status.yml — github.ref-based concurrency group on a push-triggered required-check gateway: successive main pushes can cancel the in-progress gate run (self-cancel). Worst instance of the class.
• medley/onboard-drift.yml — same class: a main push can cancel the scheduled run.
• medley/comment-review-gate.yml — no concurrency block at all (multi-event triggers; needs a per-event-safe group, not a blanket one).

Canonical block being encoded in standards#151 (corrected there from the issue text): group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} + cancel-in-progress: true for PR-scoped workflows; push/schedule runs fall through to run_id and are never cancelled. 14 workflows org-wide already conform; the three above are the real defects. Full drift list lands in the standards#151 PR body.

— placement-lane session (ref melodic-software/github-iac#78)

[kyle-sexton]

This was generated by AI during triage.

Triage disposition: needs-human · priority: medium

Rationale: architectural consolidation spanning four repos with unverified fleet-eligibility items and a premise the maintainer's own follow-up comment partially retracts — a human judgment call, not autonomous work.

The body is already a complete, research-cited design; this comment does not restate it. It records only what a human picking this up needs on top of the body: the open decisions, the blocking relationships, and confirmation against current repo state.

Confirmed against current repo state

• .github/workflows/ci.yml is itself the micro-job fan-out the issue targets: markdown / powershell / links / reference-integrity / ruff / pyright / biome / tsc … are each a separate job that checks out the repo and runs one composite from .github/actions/*.
• The consolidation primitive the first maintainer comment mandates (composite actions, not reusable workflows) already exists locally — the full .github/actions/* catalog. Consolidation is composing these as steps inside a lane, not building new primitives.
• Selector (select-runner.yml, Shape A) and fan-in gate (.github/actions/ci-status) are present and match the target shape's items 1 and 4.

Open decisions requiring human judgment (lead item first)

1. Is this still worth priority: medium+ effort now? The second maintainer comment retracts the body's core driver: per-job overhead is ~3.4s, not 20-30s, and "throughput relief shrinks once capacity supply is fixed" (ci-runner#49). If capacity is fixed, the durable case narrows to hosted minute-rounding + both-worlds consistency + starvation prevention. A human must decide whether that residual case justifies the four-repo change now. This is the reason for needs-human.
2. Primitive conflict body vs comment: the body's Sequencing says "reusable workflow/template changes here"; comment 1 says the primitive must be composite actions (reusable workflows pay full spin-up). The comment supersedes, but the body's plan text still needs reconciling before implementation.
3. Parallel-steps eligibility is unverified. Comment 1 flags background/wait "plan/runner-version eligibility unverified; test on the fleet before relying." An empirical fleet check is a prerequisite to any lane design that assumes it.
4. Worker sizing is externally gated — the "2 CPU / 4GiB consolidated-lane worker" depends on provisioning#133/docs: reconcile guard-shape wording to the enforced always() contract #134.

Blocked-by (this issue is downstream, not a blocker — no blocked-by label exists in this taxonomy, recorded here as text)

• Parent epic: melodic-software/github-iac#78 (decision record).
• Capacity supply: fix: stabilize memory-backed listener capacity ci-runner#49 — resolving it is what undercuts the throughput premise (decision 1).
• Worker sizing: melodic-software/provisioning#133 / docs: reconcile guard-shape wording to the enforced always() contract #134 (decision 4).
• Concurrency canonical block: Standard concurrency block (head_ref || run_id, cancel-in-progress) for all PR-triggered workflows standards#151 (comment 2's drift fixes; the medley workflow-structure fixes are this lane's).

Priority is medium not high: no hard deadline, and the throughput urgency that would have justified high is the very thing ci-runner#49 removes. It is downstream/gated, not a release blocker.

[kyle-sexton]

Status (2026-08-12 drain): leave open.

Next concrete step — dotfiles Shape A: confirm single-selector migration is complete on melodic-software/dotfiles .github/workflows/ci.yml (live main already has one select-runner job feeding all lanes — Shape A shape). If confirmed done, mark that sub-item complete and proceed to lane consolidation (hygiene lane collapsing micro-jobs with continue-on-error + final aggregation) and main-push concurrency collapse per the issue body.

Do not start standards/medley fan-out redesign until the dotfiles Shape A checklist item is explicitly closed out.

[kyle-sexton]

Sub-item complete (drain session): dotfiles single-selector Shape A is done on melodic-software/dotfiles main — .github/workflows/ci.yml has one select-runner job; all lanes needs: select-runner and consume needs.select-runner.outputs.runner.

This issue stays open for remaining work: lane consolidation (collapse micro-jobs into hygiene/build lanes with continue-on-error + final aggregation), main-push burst/concurrency collapse, and standards/medley fan-out redesign after that checklist item.

[kyle-sexton]

Progress slice (docs ADR, leave open)

Shape A was already confirmed done on dotfiles main. No safe structural collapse of this repo’s ci.yml micro-jobs in one PR: those jobs intentionally dogfood each .github/actions/* composite, and main-push burst concurrency (issue body item 5) still conflicts with the closed standards#151 canonical block already used here.

Shipped on branch cursor/issue-122-fanout-slice-9b36: docs/topics/ci-fanout-consolidation/ADR.md — records Shape A complete, locks the composite-action primitive, and lists ordered follow-ups.

Ordered PRs (from the ADR)

1. Human ruling on Consolidate per-run CI fan-out into few lanes; collapse main-push bursts; finish dotfiles single-selector migration #122 (needs-human): keep medium+ effort? Choose main-push burst collapse (github.ref + cancel-in-progress: false) vs standards#151 (pull_request.number || run_id).
2. dotfiles hygiene lane — collapse lint/hygiene micro-jobs with continue-on-error + step ids + final outcome aggregation; keep Shape A + ci-status.
3. dotfiles main-push concurrency — only if (1) picks burst collapse; same PR as (2) is fine.
4. Optional ci-workflows dogfood pilot — collapse one same-filter pair only (e.g. shellcheck+shfmt or ruff+pyright) + ci-status update + contract test; not a full hygiene mega-lane.
5. standards fixtures lane — fold matrix cells into an in-job fixtures lane.
6. medley last — lane consolidation + comment-2 concurrency defects (ci-status.yml / onboard-drift.yml / comment-review-gate.yml).

Issue stays open. No work on 231/232/236.

[kyle-sexton]

Frontier consensus (Fable + Sol, 2026-08-12): keep open — see drain consensus table (host / engineering / dated). No autonomous closure.

[kyle-sexton]

Drain status: ADR merged via #427. Shape A done. Remaining: human concurrency ruling + hygiene-lane consolidation PRs. Leave open.