Skip to content

Windows CI: npm-global windows-latest intermittently cancels at the global install step #3441

Closed
Closed
Windows CI: npm-global windows-latest intermittently cancels at the global install step#3441
Labels
bugSomething isn't workinginstallInstallation or packagingplatformOS/service/tray/ACL (Windows-heavy, not Windows-only)

Description

@lidge-jun
lidge-jun
opened on Sep 4, 2026

Client or integration

Other

Area

Installation or packaging

Summary

The npm-global windows-latest job in Cross-platform CI intermittently ends as cancelled at its "Install globally (downloads bundled bun)" step, roughly 8 minutes into the job. The step itself only runs for ~17 seconds before it is cancelled; every step before it succeeds, and the two steps after it are skipped, so the job is reported as a failure on the pull request without any assertion having failed.

Expected: the step either completes or fails with a diagnosable error.

Actual: the job is cancelled mid-step, which surfaces as a red required check that a rerun usually clears. On PR #3439 it happened on two consecutive runs of the same commit range, then passed on a rerun of the identical code, so it is timing-dependent rather than content-dependent.

This is CI infrastructure, not proxy behavior, but it costs a rerun cycle on otherwise-green pull requests and makes a red check ambiguous — a reviewer cannot tell a real packaging break from this without opening the job.

Reproduction

Not deterministically reproducible; it is timing-dependent. Observed occurrences on recent ci.yml runs:

  1. Run 33848227910, job 100945153338 (branch codex/260904-bug-backlog-closeout) — cancelled at "Install globally (downloads bundled bun)" after 8m15s.
  2. Run 33849960040, job 100950428875 (same branch, later commit) — cancelled at the same step after 8m12s.
  3. Run on codex/260904-muse-platform-refusal (f806eee) — npm-global windows-latest = cancelled.
  4. Run on codex/260904-windows-identity-de (7c3a542) — npm-global windows-latest = cancelled, while the same branch at b4f3fc2 and 9411090 = success.

Occurrences 3 and 4 are on unrelated branches, so this predates and is independent of the branch it was first noticed on.

Step timing from occurrence 2:

4 Install package dependencies => success  07:43:39 -> 07:50:43   (7m04s)
5 Build package assets         => success  07:50:43 -> 07:51:02
6 Pack                         => success  07:51:02 -> 07:51:06
7 Verify packed GUI asset      => success  07:51:06 -> 07:51:07
8 Install globally (bundled bun) => CANCELLED 07:51:07 -> 07:51:24
9 ocx help via bundled bun     => skipped

Note that step 4 alone takes over 7 minutes on the Windows runner, which is most of the job's budget before the cancellation lands.

Version

dev @ abdc111 (2.43.0 line)

Operating system

windows-latest GitHub-hosted runner

Provider and model

N/A

Logs or error output

# Job conclusion via the API; no assertion failure is present in the log.
npm-global windows-latest => cancelled started 2026-09-04T07:43:16Z completed 2026-09-04T07:51:28Z
   8 Install globally (downloads bundled bun) => cancelled 2026-09-04T07:51:07Z 2026-09-04T07:51:24Z
   9 ocx help via bundled bun => skipped

Individual reruns of the job are refused while the parent run is still in progress (job ... cannot be rerun), so clearing it requires gh run rerun <id> --failed after the whole run settles.

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. lidge-jun
    added
    bugSomething isn't working
    platformOS/service/tray/ACL (Windows-heavy, not Windows-only)
    on Sep 4, 2026
  2. github-actions
    added
    installInstallation or packaging
    on Sep 4, 2026

  3. lidge-jun commented on Sep 4, 2026

    @lidge-jun
    lidge-jun
    on Sep 4, 2026
    OwnerAuthor

    리뷰 · 우선순위 47 / 80

    이 이슈는 제품 프록시 버그가 아니라 Cross-platform CI의 npm-global windows-latest 잡이 중간에 cancelled로 끝나는 인프라 문제입니다. 지금 dev HEAD(5364ce01d, #3437 wp1 머지 직후)에서도 .github/workflows/ci.yml의 npm-global-smoke 잡은 timeout-minutes: 8로 고정되어 있고, 주석에도 "It is an 8-minute job"이라고 적혀 있습니다. 재현 로그를 보면 Windows에서 Install package dependencies만 약 7분 4초를 먹고, 그다음 빌드·팩·검증은 금방 끝난 뒤 Install globally (downloads bundled bun)에 들어가자마자(~17초) 잡이 취소됩니다. 스텝 자체 실패가 아니라 잡 전체 시간 한도에 걸린 패턴입니다.

    왜 이게 중요하냐면, 이 잡은 ci 집계 잡의 needs에 들어 있어서 required check처럼 빨간불이 됩니다. 그런데 로그에는 assertion 실패가 없고, 같은 커밋을 rerun하면 통과하는 경우가 많아서, 리뷰어는 "패키징이 진짜 깨졌는지 / 그냥 Windows runner가 느려서 8분을 넘겼는지"를 로그를 열어보기 전에는 구분할 수 없습니다. #3439에서 같은 커밋 범위로 두 번 연속 cancelled → 세 번째에 통과한 것도, 내용 회귀가 아니라 타이밍 의존이라는 진단과 맞습니다. 그리고 이슈에 적힌 대로 발생이 #3439뿐 아니라 muse-platform / windows-identity 등 다른 브랜치에서도 보이므로, 특정 PR 내용이 아니라 Windows hosted runner의 npm install 지연 + 8분 천장 조합입니다.

    코드 쪽을 현재 checkout 기준으로 보면 원인은 거의 한 줄에 가깝습니다. npm-global-smoke는 Ubuntu/macOS/Windows를 같은 8분 한도로 묶고, Windows만 dependency install이 7분을 넘기기 쉬운데 한도는 그대로입니다. 스텝 이름 Install globally (downloads bundled bun)이 취소되는 것은 그 스텝이 느려서가 아니라, 앞 스텝이 이미 예산을 다 써서 그 스텝 도중에 job timeout이 터진 결과에 가깝습니다. 뒤의 ocx help via bundled bun이 skipped로 남는 것도 같은 이유입니다.

    라인 741 - timeout-minutes: 8이 Windows npm install(~7분)과 겹치면 글로벌 설치 직전에 잡이 cancelled로 끝난다
    .github/workflows/ci.yml npm-global-smoke - Ubuntu/macOS/Windows가 같은 8분 한도를 공유해서 Windows만 만성적으로 예산이 모자란다
    스텝 4 Install package dependencies - Windows에서 7분+를 쓰면 남은 스텝은 사실상 0–1분뿐이라 flake가 구조적으로 난다
    ci needs의 npm-global-smoke - cancelled도 집계 실패로 올라가 required check가 빨간불이 되어 머지 판단이 흐려진다
    개별 job rerun 거부 - parent run이 진행 중이면 job만 다시 돌릴 수 없어 gh run rerun <id> --failed까지 기다려야 해서 비용이 더 커진다

    메인테이너의 판단이 필요한 지점

    너의 추천
    원인 가설이 로그·타임아웃 설정과 잘 맞으니, 먼저 Windows(또는 matrix 전체)의 npm-global-smoke timeout-minutes를 여유 있게 올리고(최소 12분 권장) 동일 브랜치에서 연속 2–3회 녹색을 확인한 뒤, 여유가 되면 Windows npm install 캐시/의존성 축소를 후속으로 잡아라. #3439의 상속 CI flake 수리와는 별 이슈로 두고, 이 이슈 번호로 CI 전용 PR 하나를 여는 편이 머지 열차에 덜 걸린다.

    이 댓글은 grok-bot이 작성했습니다

  4. lidge-jun commented on Sep 4, 2026

    @lidge-jun
    lidge-jun
    on Sep 4, 2026
    OwnerAuthor

    Root cause found, and it is not a flake — I was wrong to file it as intermittent.

    .github/workflows/ci.yml:741 sets timeout-minutes: 8 on npm-global-smoke. On the Windows runner, "Install package dependencies" alone takes about 7 minutes, which leaves under a minute for pack, verify, global install, and the help smoke. The job does not fail — GitHub cancels it at the 8-minute wall, and whichever step happens to be executing gets marked cancelled.

    That explains the detail that did not fit an install flake. Across five occurrences the cancellation always lands at 8m05s–8m15s, but on different steps: four at step 8 (global install) and the newest, on #3448, at step 7 ("Verify packed GUI asset") — a step that takes about one second and does no network I/O. Nothing about those steps is unreliable; they are simply what was running when the clock ran out.

    It looks intermittent because the Windows dependency install time varies either side of the budget. Reruns "fix" it by drawing a faster install, which is also why identical code passes on a second attempt.

    Occurrences: runs 33848227910, 33849960040, 33859671126 on codex/260904-* branches, plus codex/260904-muse-platform-refusal (f806eee) and codex/260904-windows-identity-de (7c3a542).

    The comment in the job says "It is an 8-minute job, so there is nothing to win by moving it" — that assumption is what is now stale on the Windows leg. Raising the timeout is the correct fix rather than a workaround; the job is genuinely slower than its budget on one OS. Opening a PR.

  5. lidge-jun
    added a commit that references this issue on Sep 4, 2026

    ci(npm-global): give the Windows leg a timeout it can meet (#3455)

    9742832

  6. lidge-jun commented on Sep 4, 2026

    @lidge-jun
    lidge-jun
    on Sep 4, 2026
    OwnerAuthor

    Fixed by #3455, squash-merged into dev as 9742832.

    npm-global-smoke goes from timeout-minutes: 8 to 20. The job was never failing — it was being cancelled at the wall, because installing dependencies on the Windows runner alone takes about 7 of the 8 minutes. Whichever step happened to be executing when the clock expired got marked cancelled, which is why it appeared on the global install four times and on a one-second asset check once, and why a rerun of identical code always passed.

    tests/ci-workflows.test.ts pins each job's timeout by name so values cannot be swapped between jobs unnoticed, so it failed on the first push of the fix — correctly. The guard expectation moved with the workflow in the second commit, with the reason recorded next to the note that already explains the same lesson for the Windows test shards.

    Closing manually: the PR targeted dev, and GitHub only auto-closes linked issues on the default branch.

    My original filing called this intermittent, which was wrong — the cause is deterministic and the variance was just Windows install time landing either side of an 8-minute budget. The step-timing comparison across five occurrences is what made that visible.

  7. lidge-jun
    closed this as completedon Sep 4, 2026
  8. agentHits
    added a commit that references this issue on Sep 17, 2026

    ci(npm-global): give the Windows leg a timeout it can meet (lidge-jun…

    27da80c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinginstallInstallation or packagingplatformOS/service/tray/ACL (Windows-heavy, not Windows-only)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions