Skip to content

ci: scheduled runs queue instead of cancelling each other in the nightly concurrency groups - #3368

Merged
ranxianglei merged 2 commits into
masterfrom
fix/ci-cron-livelock
Oct 16, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
fix/ci-cron-livelock

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

ai: zhipuai-lb/glm-5.3

背景(诚实声明)

我在 #3331 评审里把这条标成 HIGH「CI 总是失败的直接原因」——实测数据推翻了这个断言:查了 10 条 30 分钟 cron lane 的近 30 次 scheduled 运行,0 次取消、全部成功,实际用时都在几分钟内(45min 只是 timeout 上限,不是真实用时)。所以这是潜在风险加固,不是现行故障修复。特此更正。

潜在风险是什么

每个 e2e/registry/windows lane 的 concurrency 组都是这个形状:

concurrency:
  group: e2e-${{ github.event_name == 'schedule' && 'nightly' || github.ref }}
  cancel-in-progress: true

注释写的意图(#2637 plan #3)是「push 不能取消当天的 nightly、同 ref 的推送互相顶替」。但 GitHub 的 cancel-in-progress 对组内也生效:同一个 workflow 的两条 scheduled 运行落在同一个 *-nightly 组里,后到的会杀掉在跑的。

一旦哪天某次 scheduled 运行真的超过 30 分钟(慢上游、真实 codex 抖动):

timeout-minutes: 45 (ci-e2e.yml) > 30 分钟 cron 间隔,说明设计上就允许运行超过间隔——这两个设置目前互相矛盾。

修复

cancel-in-progress 只对非 schedule 事件生效:

cancel-in-progress: ${{ github.event_name != 'schedule' }}
  • PR/push 的「新推送顶替旧运行」语义逐字不变;
  • scheduled 运行改为排队:在跑的不动,新到的排队(最多 1 条 pending,再新的会顶掉 pending——pending 没启动,不花 token);
  • 跑完的那条能记绿,gate 正常学到 head 已覆盖。

10 个文件同一行改动 + 注释(全部带 nightly 组的 lane;ci-theorem.yml 无 concurrency 组,不涉及)。

验证

  • YAML 全量解析通过(js-yaml);
  • 改动仅 concurrency 块,job 定义零变化;
  • 当前历史数据(0 取消)证明不改变现状行为——只是把超时长运行从「必死」变成「能活」。

Refs #3331 (F-1)

The nightly concurrency groups ('schedule' -> shared 'nightly' suffix)
carry cancel-in-progress: true, and GitHub applies that WITHIN the group:
two scheduled runs of the same workflow kill each other. The per-file
comments ('keeps scheduled runs out of that cancel scope', #2637 #3)
describe the intent — pushes must not cancel the nightly run — but the
flag as written also lets the 30-min cron kill its own predecessor if a
run ever outlives the interval: tokens spent, no green recorded, and the
smart gate (#3295) can never learn the head is covered.

Empirically no lane has hit this yet (current scheduled runs finish in
minutes, 0 cancellations in recent history) — this is latent hardening,
not a live-fire fix. cancel-in-progress is now scoped to non-schedule
events, so same-ref supersede semantics are unchanged while scheduled
runs queue behind each other (at most one pending; a superseded PENDING
run costs nothing).
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: fix/ci-cron-livelock (7699263)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3972-pr.3368.3233e37

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3368.tgz
npm install -g package

Install instructions are refreshed on each push.

…lanes (#3360)

The original change covered the ten e2e/registry/windows-codex lanes but left
ci.yml, ci-image.yml and ci-zcode-real.yml on cancel-in-progress: true — the
same within-group mutual cancellation applies to their scheduled ticks too
(ci-image additionally shares one group across event types, so a dispatch
could cancel a running cron run as well). Same one-line conditional, same
comment, no other changes.
@ranxianglei
ranxianglei merged commit 8fad9c6 into master Oct 16, 2026
46 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.490-master.8fad9c6 (rolling master tag).

npm install -g billion-context@0.0.490-master.8fad9c6  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant