Skip to content

test(cli): measure cold status setup before timed projections - #4948

Merged
lidge-jun merged 1 commit into
devfrom
codex/lane-g-2590-status-cold-start
Sep 17, 2026
Merged

lidge-jun merged 1 commit into
devfrom
codex/lane-g-2590-status-cold-start

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Sep 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

Separate the cold full-CLI status path from the version-projection assertions. Windows run 35277467396 killed the first status child after 15,587 ms; the next row took 13,074 ms and subsequent two-child rows took 1,518–1,741 ms. The first row was measuring cold setup plus behavior while later rows measured the warmed path. A later passing run does not resolve that defect.

One Windows beforeAll fixture now executes the same real status and health path before the matrix. Setup must succeed and is bounded by the existing 45-second spawn budget; its child has 25 seconds, leaving the existing 15-second Windows tree-removal bound plus five seconds for reap. All sixteen original JSON/human children retain their 15-second deadlines and result assertions. Cold setup and individual child timings are logged for hosted evidence. No retry, global budget increase or platform skip is added.

The observed cold-path dependence does not isolate Bun transpilation, Defender, filesystem caches or external diagnostic probes. This change makes that setup cost explicit and measured without claiming an unobserved subsystem cause.

Verification

  • Local verification was NOT RUN because this lane forbids local suites, focused tests, typecheck, builds, installs and proxy execution. Hosted CI is the executable verification.
  • Original failure: run 35277467396, Windows 7/9 job 105391449887, cli-status-json.test.ts:77.
  • Static review retained every original output assertion and both formats for all eight rows; child reap, server shutdown and temporary-home cleanup remain mandatory.
  • Independent review required enough teardown margin; the setup child allowance was narrowed from 40 to 25 seconds before publication.
  • Exact base: a0f611d. The release host will re-freeze and execute the combined tree after integration.
  • Exact head cca06b1 has all nine Windows shards and all four Linux shards successful in full dispatch 35279223062, with gates green. Windows job 105397453194 recorded setup at 1755 ms and all sixteen measured children at 583–933 ms.
  • The release owner explicitly authorized integration on that targeted evidence while macOS combo-failover teardown is investigated separately. PR run 35279139497 has one macOS shard-2 failure in the unrelated connect-cancellation hook (job 105396939652; 12807 pass, 12 skip, 1 fail). This is an attributed integration exception, not a claim that the full macOS workflow is green; remaining macOS jobs and the final merged-tree run are separate evidence.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed. The fixture documents where the real cold start is measured; the audit record tracks the hosted evidence.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. Only synthetic fixture values and elapsed times are logged; isolated paths and teardown are retained.

@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the chore Maintenance, CI, tests, refactors, or build changes (not a user-facing bug or feature). label Sep 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@lidge-jun
lidge-jun marked this pull request as ready for review September 17, 2026 22:21
@lidge-jun

Copy link
Copy Markdown
Owner Author

Merging this on a recorded exception, so the reasoning is in the record rather than in a thread.

The evidence this change needs is Windows, and it has it. At exact head cca06b1693fc4f7d4711bb920ea5f5417655edf5, dispatch run 35279223062 has all nine Windows shards green, Linux 1/4 through 4/4 and gates are green in both that run and the PR-event run 35279139497, and the sixteen timed projection assertions pass with the setup child at 1755ms and every timed child between 583ms and 933ms. That is the fixture doing exactly what it was changed to do: the cold cost moved into an explicit setup outside the measured window, and every assertion now measures the same warm path against the same unchanged 15s bound.

The one red is macos 2/2, and it is not this change. It is server combo failover 030 activation matrix > connect cancellation wins with 499, no backup, warning, or cooldown timing out in a beforeEach/afterEach hook at 30066ms, in a file this branch does not touch. A dedicated lane owns that investigation together with two longer silent hangs on sharded macOS legs at the same tip. Holding a Windows fix hostage to an unrelated macOS teardown defect would delay the release without making anything safer, and attributing that failure to this PR would be false.

Recording the boundary explicitly: this PR is merged with Windows and Linux evidence at its exact head, with a known unrelated macOS failure outstanding and owned elsewhere. It is not a claim that macOS is green at this SHA, and the release candidate is still held for exactly that reason.

@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner September 17, 2026 22:21
@lidge-jun
lidge-jun merged commit 61ee647 into dev Sep 17, 2026
57 of 60 checks passed
@lidge-jun
lidge-jun deleted the codex/lane-g-2590-status-cold-start branch September 17, 2026 22:21

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cca06b1693

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// Keep this bounded and observable: a failed warmup is a setup failure, never a retry.
// The 45s hook reserves removeTreeWithRetry's existing 15s Windows cleanup bound plus 5s
// to reap the child, leaving 25s for the cold status invocation itself.
const result = await runTimedStatus(home, codexHome, true, 25_000);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use the Windows cold-spawn budget for the warmup

On a Windows runner where the first Bun child needs more than 25 seconds, this setup still kills the cold child and fails the entire describe block before any 15-second projection assertion runs. The repository already defines COLD_SPAWN_BUDGET_MS as 90 seconds on Windows because a first proxy child has taken about 50 seconds in CI, so the new warmup merely moves the same cold-start flake from 15 to 25 seconds. Give this one setup invocation the cold-spawn budget and enlarge the enclosing hook enough to retain the cleanup margin, while leaving the measured projection children at INTERNAL_DEADLINE_MS.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-17T22:25:05.617577Z cca06b1 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This was referenced Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Maintenance, CI, tests, refactors, or build changes (not a user-facing bug or feature).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant