melo-lap-001 runner containers missing node since 2026-08-13 — github-iac governance-verify lane red daily

[kyle-sexton]

Summary

github-iac's scheduled governance-verify lane has failed every day since 2026-08-13 with
node: command not found (runs 31684071334 → 32009922779+). The job runs
node .github/governance-verify/verify-org-security.cjs directly on the runner image (no
setup-node by design — the dual-token invocation avoids github-script), and jobs landing on
ci-runner-melo-lap-001--* containers (label melodic-ubuntu-24.04-x64) no longer find node
on PATH. The failure is infrastructure, before any assertion — the org's daily drift backstop
over expected-org-security.json (fork policy, runner groups, visibility, SHA-pin, workflow
permissions) has been asserting nothing for 5+ days.

Evidence

• /home/runner/_work/_temp/<id>.sh: line 3: node: command not found on
  Verify live org security state, runner ci-runner-melo-lap-001--dkr5tjr68utc-ca (2026-08-17).
• Other lanes pass when scheduled onto the desk runner, so the regression looks specific to the
  melo-lap-001 image/environment.
• Timing note: 2026-08-13 is also when github-iac's CI_RUNNER_POLICY org variable's visibility
  flipped out-of-band (unidentified mutator, tracked as audit task T9) — possibly the same
  infrastructure event.

Requested

Restore node on the melo-lap-001 runner image (or its PATH for job containers), and consider an
image smoke test that fails the build when the toolchain contract (node, jq, etc.) regresses.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TMAsSUZ5YnaQDLBvTFiXW7

[kyle-sexton]

In-repo slice is complete in #293. Consensus (Fable + Sol): do not bake Node into the worker image. scripts/verify-worker-image.sh now fails if node/npm/npx appear on PATH.

This environment cannot edit github-iac. Durable fix, still required, no new tracking issue:

• In the governance-verify job that shells node .github/governance-verify/verify-org-security.cjs, add a pinned actions/setup-node step before that invocation (same pin style as other Actions in that repo).
• Do not restore Node on the ci-runner worker image.

Leave this issue open until that github-iac edit lands.

[kyle-sexton]

This was generated by AI — automated lane telemetry.

Triage lane claiming this item for the current cycle (instance MELO-LAP-001-triage-202608230449). Lease TTL 60min, expires 2026-08-23T05:59:50Z. Dispatching /work-items:triage 286.

[kyle-sexton]

This was generated by AI during triage.

Triage outcome: human-gated (cross-repo capability blocker)

Verification — confirmed, still current. Governance live verify in melodic-software/github-iac has failed on every scheduled run through 2026-08-22 (10 consecutive days, runs 31684071334 → 32560996882), same node: command not found failure as this issue's evidence. The job still shells node .github/governance-verify/verify-org-security.cjs directly with no actions/setup-node step present in the workflow file (checked live). Root cause matches the original report exactly.

In-repo remediation — complete. #293 merged 2026-08-21: scripts/verify-worker-image.sh now fails the worker-image build if node/npm/npx are on PATH. Per the Fable/Sol consensus recorded above, Node will not be restored on the melo-lap-001 worker image — the original "Requested" section's first bullet is superseded by that decision.

Remaining work is cross-repo (github-iac) and outside this lane's write scope. This triage lane operates only on melodic-software/ci-runner and cannot edit melodic-software/github-iac. Checked github-iac issues and PRs (open + merged) for an existing tracking item or an in-flight fix — none reference setup-node or otherwise touch the governance-verify job's toolchain step; the workflow file itself still has no Node-provisioning step. Routing to human-gated rather than closing, per the standing direction above to leave this issue open until the github-iac edit lands.

No ci-runner-scoped lane can move this further. This session has no coordinator lane for melodic-software/github-iac, and the standing direction above explicitly declines a companion tracking issue there. Until a human (or a github-iac-scoped lane) picks this up directly, this item has no autonomous progress path from ci-runner by construction — that is the intended effect of human-gated here, not an oversight.

Agent Brief

Type: Bug
Summary: The Governance live verify scheduled workflow in github-iac invokes Node directly without provisioning a Node toolchain, so every run fails before the org-security drift assertion executes.

Current behavior:
The governance-verify job's org-security-verification step invokes a Node script (verify-org-security.cjs) directly via a bare node shell invocation, relying on Node being present on the runner's PATH. Since the ci-runner worker image no longer provides Node (deliberate — see above), the step fails immediately with node: command not found, and the daily drift assertion over the org's security posture (fork policy, runner groups, visibility, SHA-pin, workflow permissions) never runs.

Desired behavior:
The job provisions its own Node toolchain via a pinned actions/setup-node step (matching the SHA-pin style used by other Actions steps in that repo) before invoking the verification script, so the job no longer depends on the worker image providing Node. The dual-token, single-process invocation style (avoiding github-script) is preserved — only toolchain provisioning changes.

Key interfaces:

• The governance-verify workflow's org-security-verification job — add a pinned Node-setup step ahead of the existing script-invocation step.
• No change to verify-org-security.cjs itself or to its dual-token (GOVERNANCE_TOKEN/OBSERVER_TOKEN) invocation contract.

Acceptance criteria:

• The next scheduled Governance live verify run completes the org-security-verification step without a node: command not found (or equivalent PATH) failure.
• The Node-provisioning step is pinned by SHA, consistent with this repo's other Actions references.
• The script's existing dual-token invocation and assertions (fork policy, runner groups, visibility, SHA-pin, workflow permissions) are unchanged and continue to run against live org state.

Out of scope:

• Restoring Node on the melo-lap-001 / ci-runner worker image — explicitly rejected (see test(worker-image): reject workflow Node on the worker PATH #293 and the consensus recorded above).
• Adding an image-level toolchain smoke test to ci-runner (a secondary suggestion in the original report) — track separately if wanted; not required to close this item.
• Any change to verify-org-security.cjs's assertions or output format.

---

This item stays open on ci-runner per the standing direction above, tracking the github-iac-side fix. A human (or an agent lane with github-iac write access) should apply the brief above there and close this issue once the scheduled run is green.

[kyle-sexton]

Resolved in github-iac: melodic-software/github-iac#360 merged as 1a6aa56 (SHA-pinned actions/setup-node@8207627 v7.0.0 before the verifier step). Fresh-context verifier PASS on all four criteria; on the PR's own run the step provisioned Node on the Node-less melo-lap-001 worker image. First scheduled-equivalent run on main after merge: https://github.com/melodic-software/github-iac/actions/runs/33074326741 — success. Node stays off the worker image per the decision recorded above.