feat(runners): route on fleet liveness, drop rerun-to-hosted branch

[kyle-sexton]

Part of melodic-software/github-iac#79 (Epic melodic-software/github-iac#78, Phase 1 W1).

What

• Liveness routing: the prefer-self-hosted candidate test drops busy === false; any matching managed-prefix runner with status == "online" routes the workflow self-hosted. GitHub natively queues jobs targeting a busy fleet and only fails them after 24 hours queued (routing precedence docs). Hosted fallback now happens only when the fleet is fully offline.
• Delete the rerun→hosted hack: removed the run_attempt > 1 → hosted branch, the RUN_ATTEMPT input, and the workflow-level github.run_attempt == 1 token-mint guard. A re-run reuses the previous attempt's successful selector output (re-run docs), so the branch only converted re-runs into paid hosted work jobs.
• Honest naming: idle-runner-count → online-runner-count; reasons idle → online, no-idle-runner → no-online-runner. busy is no longer consumed, so it left inventory validation.
• Vendored bundle regenerated (render-select-runner-workflow.cjs --check green); README selector contract updated, including one-preflight-job-per-workflow guidance.

Not in this PR (sequenced follow-ups)

• local-runner-canary.yml / production-ha-proof.yml stay pinned to the reviewed old selector SHA and stay self-consistent (they assert idle semantics of that pinned revision). They migrate in their own pin-bump PR once this merges.
• standards runner-policy SHA allowlist bump + consumer-repo fan-out collapse land as separate PRs after merge (allowlist keys on this PR's merge SHA).

Verification

• node --test .github/scripts/*.test.cjs: 169/169 pass, including new cases: busy online runner routes self-hosted; busy empty-label scale-set runner keeps the inferred route; fully offline fleet routes hosted with no-online-runner.
• Empirical routing acceptance (fleet online → ci-runner-melo-*, fleet offline → hosted, re-run keeps route) runs on a routed private repo after the consumer pin bump, per issue feat: support policy-only Pulumi deploy guard #79.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EPDbXgonTuFwFwdTtHaCmw

---

Note

Medium Risk
Changes adaptive CI runner routing for all prefer-self-hosted consumers—busy fleets stay on self-hosted instead of falling back to paid hosted minutes—while removing rerun-forced hosted routing; behavior is well covered by unit tests but rollout affects org-wide workflow execution.

Overview
prefer-self-hosted now keys off fleet liveness, not idle capacity. Matching managed-prefix runners with status === "online" route self-hosted even when busy is true; hosted fallback applies only when no eligible runner is online (no-online-runner), relying on GitHub to queue jobs on a busy fleet.

The rerun→hosted shortcut is removed: no run_attempt > 1 preflight branch, no RUN_ATTEMPT env/input, and the observer-token step no longer gates on github.run_attempt == 1. The rerun reason and idle-centric API are gone—idle-runner-count / idle / no-idle-runner become online-runner-count / online / no-online-runner, and runner inventory validation no longer requires busy.

Tests, vendored select-runner.yml, selector-conformance.yml, and README are updated to match (including busy-online and fully-offline cases, and public-guard expectations without attempt-2 rerun).

^{Reviewed by Cursor Bugbot for commit d1e14eb. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_22d40f7c-a1b3-4602-8a86-02e22e85c4d4)

[claude]

Claude finished @kyle-sexton's task in 2m 29s —— View job

---

PR Review

• Gather context and understand the request
• Review select-runner.cjs — core logic changes
• Review select-runner.test.cjs — test coverage
• Review select-runner.yml — workflow changes
• Review README.md — documentation accuracy
• Post findings

---

Summary

Solid, well-scoped PR. The liveness-over-idleness routing change is correct and the implementation is internally consistent. The rerun-to-hosted removal is well-justified, and the refactoring is thorough across all layers (source, vendored bundle, tests, docs).

---

Findings

1. validateRunner drops busy check without acknowledging the silent-pass widening

ci-workflows/.github/scripts/select-runner.cjs

Lines 266 to 267 in 83f90e1

	typeof runner.status !== "string" ||
	(Object.hasOwn(runner, "ephemeral") &&

The old schema rejection typeof runner.busy !== "boolean" would throw InvalidResponseError (→ safe hosted fallback) if the API ever emitted a non-boolean busy. Removing it means a malformed busy field silently passes. Since busy is intentionally unused now this is not a routing concern, but it is a subtle narrowing of the validation surface. No action required unless you want a comment noting that busy is deliberately unvalidated rather than merely forgotten.

2. README re-run claim is accurate for one scenario but imprecise for another

https://github.com/melodic-software/ci-workflows/blob/83f90e179d8d664e5f9858066242a74c03382a94/README.md#L787-L791

"A re-run reuses the prior attempt's successful selector output, so it follows the original routing decision."

This is true for Re-run failed jobs (the selector, if successful on attempt 1, is skipped and its outputs are inherited). It is not true for Re-run all jobs — GitHub re-runs the selector fresh in that case. Since self-hosted-only callers re-run the selector regardless, the current sentence could mislead a consumer into thinking routing is always deterministic across re-runs. The guidance to call the selector exactly once per workflow (added elsewhere in this PR's README update) largely covers this, but the re-run sentence itself over-promises. Consider qualifying: "Re-running only failed jobs reuses the prior attempt's successful selector output…"

3. Token-mint guard removal — implicit coupling

ci-workflows/.github/workflows/select-runner.yml

Lines 83 to 84 in 83f90e1

	if: >-
	inputs.policy == 'prefer-self-hosted' &&

Removing github.run_attempt == 1 from the observer-token if guard is correct (re-runs now do real selection), but the old guard was the only mechanism ensuring the token was not consumed on every re-run attempt under the old hack. Now that the guard is gone, the App token is minted on every attempt — including "re-run all jobs" where the selector runs fresh. This is the right behavior, just worth confirming the App token has no rate-limit or audit concern at re-run frequency. No code change needed; noting for record.

4. Test split of "empty-label scale-set inference" — ✅ correct and complete

ci-workflows/.github/scripts/select-runner.test.cjs

Lines 374 to 406 in 83f90e1

	},
	);
	assert.equal(observed.route, "GET /repos/{owner}/{repo}/actions/runners");
	assert.equal(observed.parameters.owner, "kyle-sexton");
	assert.equal(observed.parameters.repo, "medley");
	assert.equal(result.route, "self-hosted");
	});
	test("filter requires exact label, managed prefix, and online state", async () => {
	const inventory = [
	runner({ name: "unmanaged-1", labels: [{ name: "unrelated-label" }] }),
	runner({ status: "offline" }),
	runner({ labels: [{ name: "melodic-ubuntu-24.04-x64-other" }] }),
	runner({ name: "ci-runner-melo-lap-001-1" }),
	];
	const result = await selectRunner(input(), {
	request: async () => response(inventory),
	});
	assert.equal(result.route, "self-hosted");
	assert.equal(result.onlineRunnerCount, 1);
	});
	test("a busy online runner keeps the self-hosted route so GitHub queues the job", async () => {
	const result = await selectRunner(input(), {
	request: async () => response([runner({ busy: true })]),
	});
	assert.deepEqual(result, {
	runner: "melodic-ubuntu-24.04-x64",
	route: "self-hosted",
	reason: "online",
	onlineRunnerCount: 1,
	});
	});

The old test mixed offline + busy assertions into one case. The split into "…requires the managed prefix and online state" (offline → hosted) and "a busy empty-label scale-set runner keeps the inferred single route live" (busy → self-hosted) is exactly right. Each test now expresses a single invariant.

5. "stable saturation routes hosted" → "a fully offline fleet routes hosted" rename — ✅

ci-workflows/.github/scripts/select-runner.test.cjs

Lines 517 to 525 in 83f90e1

	}),
	]),
	});
	assert.equal(result.route, "hosted");
	assert.equal(result.reason, "no-online-runner");
	assert.equal(result.onlineRunnerCount, 0);
	});
	test("a busy empty-label scale-set runner keeps the inferred single route live", async () => {

The old test was semantically wrong after the behavior change (it tested busy: true → hosted, which now means self-hosted). The replacement correctly uses status: "offline" to trigger no-online-runner. Good catch.

6. Output contract — breaking change acknowledged

idle-runner-count → online-runner-count and reasons idle/no-idle-runner → online/no-online-runner are breaking output changes. The PR explicitly notes that existing callers pin the old SHA and will migrate separately. The selector-conformance.yml reference in the README correctly no longer mentions the attempt-2 contract. LGTM.

---

Verdict

Approve. No correctness bugs, no security regressions, no CLAUDE.md violations. The one documentation nuance (finding 2) is minor and worth a pass on if the team prefers brevity. Everything else is clean.

[cursor]

Stale comment

Risk: medium. Cursor Bugbot did not complete successfully (check skipped due to usage limit), so I cannot approve on automated review signals. Human review is needed; no reviewers were assigned because no eligible reviewers exist beyond the PR author in this repository.

  

_{Sent by Cursor Approval Agent: Pull Request Router and Approver}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 83f90e179d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_77d97d02-10de-420c-a513-a71929b914ec)

[claude]

Claude finished @kyle-sexton's task in 2m 28s —— View job

---

PR Review

• Read changed files
• Review select-runner.cjs — core logic
• Review select-runner.test.cjs — test coverage
• Review select-runner.yml — workflow changes
• Review selector-conformance.yml — conformance contract
• Review README.md — documentation accuracy
• Post findings

---

Summary

Well-scoped, internally consistent PR. The liveness-over-idleness change is semantically correct and the rerun hack removal is well-justified. All five layers (source, vendored bundle, tests, conformance, docs) move in lockstep. No correctness bugs, no security regressions, no CLAUDE.md violations.

One prior review comment to address: the Codex P2 finding about selector-conformance.yml still expecting attempt > 1 is not valid against this PR. The old dynamic expression ${{ github.run_attempt > 1 && 'rerun' || 'hosted-only' }} was removed and replaced with the literal hosted-only string. The concern is already resolved.

---

Findings

1. validateRunner drops busy — correct, but worth noting the intentional widening

ci-workflows/.github/scripts/select-runner.cjs

Lines 256 to 281 in d1e14eb

	function validateRunner(runner) {
	if (runner === null || typeof runner !== "object") {
	throw new InvalidResponseError(
	"runner inventory contains a malformed runner",
	);
	}
	if (
	!Number.isInteger(runner.id) ||
	typeof runner.name !== "string" ||
	!exactNonEmptyString(runner.os) ||
	typeof runner.status !== "string" ||
	(Object.hasOwn(runner, "ephemeral") &&
	typeof runner.ephemeral !== "boolean") ||
	!Array.isArray(runner.labels) ||
	runner.labels.some(
	(label) =>
	label === null ||
	typeof label !== "object" ||
	!exactNonEmptyString(label.name),
	)
	) {
	throw new InvalidResponseError(
	"runner inventory contains a malformed runner",
	);
	}
	return runner;

The old typeof runner.busy !== "boolean" check would throw InvalidResponseError (safe hosted fallback) if the API emitted a non-boolean busy. Since busy is now intentionally absent from the filter, coupling validateRunner to it would create false schema coupling — the removal is correct. The remaining typeof runner.status !== "string" check still ensures the liveness gate field exists and is typed. No change needed, just confirming this is a deliberate and sound narrowing of the validation surface, not an oversight.

2. Re-run semantics in README are now correctly scoped

ci-workflows/README.md

Lines 158 to 160 in d1e14eb

	fleet falls back to the hosted route. Re-running failed jobs reuses the
	prior attempt's successful selector output; re-running all jobs makes a
	fresh liveness decision. Neither forces the hosted route.

"Re-running failed jobs reuses the prior attempt's successful selector output; re-running all jobs makes a fresh liveness decision. Neither forces the hosted route."

The previous Claude review (finding 2) flagged that the re-run sentence over-promised. This PR's README now correctly distinguishes the two re-run modes. The text is accurate: "re-run failed jobs" skips the selector and inherits its previous outputs; "re-run all jobs" re-executes the selector, which now makes a fresh liveness decision rather than being forced to hosted. ✅

3. Observer-token guard removal — correct and not a security concern

ci-workflows/.github/workflows/select-runner.yml

Lines 83 to 101 in d1e14eb

	if: >-
	inputs.policy == 'prefer-self-hosted' &&
	(inputs.self-hosted-label != '' || inputs.self-hosted-labels-json != '') &&
	inputs.hosted-runner != '' &&
	(inputs.scope == 'organization' || inputs.scope == 'repository') &&
	(inputs.scope != 'repository' || github.event.repository.name != '') &&
	inputs.managed-runner-prefix != '' &&
	inputs.observer-client-id != '' &&
	inputs.api-timeout-seconds >= 1 &&
	inputs.api-timeout-seconds <= 60 &&
	(github.event_name == 'push' ||
	github.event_name == 'schedule' ||
	github.event_name == 'workflow_dispatch' ||
	(github.event_name == 'pull_request' &&
	github.event.pull_request.head.repo.full_name == github.repository)) &&
	env.HAS_OBSERVER_SECRET == 'true' &&
	github.event.repository.private == true &&
	github.actor != 'dependabot[bot]' &&
	github.secret_source != 'Dependabot'

Removing github.run_attempt == 1 from the token-mint if condition is correct. That guard was an efficiency mechanism (avoid minting a token on re-runs that would be routed hosted anyway), not a security gate. The actual security gates — event type allowlist, github.event.repository.private, fork PR check, dependabot[bot] exclusion — are all unchanged and still guard token issuance before the action can execute. Re-runs now mint the token and make a real liveness decision, which is the intended behavior. ✅

4. selector-conformance.yml — EXPECTED_GUARDED_REASON simplified correctly

ci-workflows/.github/workflows/selector-conformance.yml

Line 116 in d1e14eb

EXPECTED_GUARDED_REASON: hosted-only

Old: ${{ github.run_attempt > 1 && 'rerun' || 'hosted-only' }} (dynamic)
New: hosted-only (literal)

The conformance PR runs from a fork/public context where the public-repository guard fires before any inventory check, so hosted-only is always the expected reason regardless of attempt number. The dynamic expression was the forward-coupling to the now-deleted rerun path; its removal is correct. ✅

5. self-hosted-only + rerun — no behavior change, as expected

The old runAttempt > 1 check in preflight was explicitly guarded by !selfHostedOnly, so self-hosted-only was never affected by it. Removing runAttempt from the input type and environment wiring does not change self-hosted-only routing behavior. ✅

6. Test coverage for the behavioral change

ci-workflows/.github/scripts/select-runner.test.cjs

Lines 310 to 320 in d1e14eb

	});
	test("self-hosted candidate cannot equal the configured hosted runner", async () => {
	const result = await selectRunner(
	input({ selfHostedLabel: "Ubuntu-24.04" }),
	{
	request: requestMustNotRun,
	},
	);
	assert.equal(result.route, "hosted");
	assert.equal(result.runner, "ubuntu-24.04");

New test "a busy online runner keeps the self-hosted route so GitHub queues the job" directly asserts the core behavior change. The split of the scale-set inference test into offline → hosted and busy → self-hosted cases is exactly the right split — each test now expresses a single invariant. Renamed "stable saturation routes hosted" → "a fully offline fleet routes hosted" with status: "offline" fixes a test that was semantically wrong after the behavior change. Coverage looks complete. ✅

7. Vendored bundle sync — verified

The inline bundle in select-runner.yml (lines 185–769 of the workflow) mirrors the source in select-runner.cjs identically. The integration test "generated github-script bundle executes the tested adapter" asserts end-to-end that the bundle runs and emits reason: "online" / online-runner-count: "1". ✅

---

Verdict

Approve. No blocking issues. The liveness routing change is correct, the rerun hack removal is justified, and the implementation is thorough across all layers. The Codex P2 concern about conformance is already resolved in this PR.

[cursor]

Risk: medium. Cursor Bugbot did not complete successfully (check skipped due to usage limit), so I cannot approve on automated review signals. Human review is needed; no reviewers were assigned because no eligible reviewers exist beyond the PR author in this repository.

  

_{Sent by Cursor Approval Agent: Pull Request Router and Approver}