Repository navigation
fix(e2e): make the nightly e2e suites green - #2256
Merged
Merged
Conversation
The harness could not reliably kill wrangler: walking the process tree with pgrep left a window in which wrangler respawned workerd, and the orphan kept the test's fixed port. Spawn wrangler detached and SIGKILL the process group instead, sharing one helper across the Think e2e files. With the harness fixed, three real problems surfaced: Think's in-flight-turn probes only counted cf_agents_runs although chat turns run on Tasks; the post-persist failure test asserted onError is never called, which Tasks now does; and a Think turn interrupted before its first chunk was classified as a continuation rather than a retry, so agent-tool parents sealed the child as errored. The ai-chat work-budget agent banked no work under the stream-derived work marker and now flushes a full segment. Also fixes the broken shell quoting in the Playwright version step, which made the cache key empty in every run.
🦋 Changeset detectedLatest commit: 499876f The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
⚪ agents import sizesMeasured 340 runtime imports as minified bundles. The primary size is gzip; raw minified size is included for diagnosis. An existing import growing by more than 10% is marked red. This report is informational.
Compared No import sizes changed. All 340 current runtime imports
Reported by agent-think[bot]. |
agents
@cloudflare/ai-chat
@cloudflare/codemode
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why the nightly has been red
The Nightly E2E workflow has failed every night since Aug 24 (last green Aug 23). Three jobs fail; the diagnosis below is from running each failing file locally against an untouched
maincheckout and comparing against the fix.1.
E2E: think (chat recovery)andE2E: agents (fiber eviction): the harness could not actually kill wranglerEvery Think e2e file and the agents e2e helpers killed
wrangler devby walking the process tree withpgrep -Pand SIGKILLing children first. Eachpgrepis a synchronous spawn that takes milliseconds; in that window wrangler, still alive, saw workerd die and respawned it. The respawned workerd was not in the snapshot, survived the walk as an orphan (parent pid 1), and kept listening on the test's fixed port. Symptoms in every red run:Address already in use (127.0.0.1:1879x)on the restart,Updated and readyprinted by a process the test believed dead,Port 1879x did not free in time, and "interrupted" turns that simply finished on the old server so nothing was ever recovered.Fix: spawn wrangler
detached: true(a process-group leader) and SIGKILL the group, which takes downnpm exec, wrangler, esbuild and every workerd in one signal. This is what the ai-chat harness already did. The 14 Think copies of the helper are replaced by one sharedpackages/think/src/e2e-tests/wrangler-process.ts; the agents copies are fixed in place.With the harness fixed, three real problems surfaced underneath:
hasFiberRowsprobes only countedcf_agents_runs. Chat turns run on the Tasks capability (cf_agents_task_runs) since refactor(chat): run recovery continuations on Tasks #2194, so "is a turn in flight" answered no. The probes now count both engines, as ai-chat's already did.onErroris not called. Since chat turns are Tasks runs, a failed turn is reported throughonError(Agent wiresonErrorinto Tasks). The test now expects the failure in theonErrorlog as well as the chat error frame; the gap it documented is closed.@cloudflare/thinkpatch changeset). Think opens the resumable stream row before inference, so such a turn has a stream id, no persisted partial, and a user message as its latest leaf._recoverablePreStreamUserIdreturned null as soon as a stream id existed, the turn was classifiedcontinue,continueLastTurnfound no assistant message and marked the incidentskipped, and a parent tailing that child through agent tools sealed the run aserror: Agent tool run was interrupted before the child could finish.The stream id is no longer treated as evidence of content; the absence of a persisted partial plus a leaf that is still the turn's user message decides retry, mirroringAIChatAgent's empty-partial new-turn rule (AIChatAgent orphaned stream recovery can merge a new assistant response into the previous assistant message #1691).2.
E2E: ai-chat (SIGKILL recovery): the work-budget exhaustion agent relied on the old work counterChatWorkBudgetExhaustAgentemitted a singletext-startand hung, relying on the KV work counter that credited a chunk at production time. Since #2223 the work meter is derived from the stream log, where a chunk counts once its packed segment (CHUNK_BUFFER_SIZE, ten chunks) is flushed, so the agent banked zero work and a zero budget could never be exceeded. The agent now emits exactly one segment's worth of chunks before hanging.3.
E2E: agents (browser connector): broken shell quoting in the workflowInside single quotes bash passes
\"through literally, so node seesrequire(\"...\")and throwsSyntaxError: Invalid or unexpected token. The step still exits 0 (theechosucceeds with an empty version), so it never failed the job, but the Playwright cache key has been empty in every run. Fixed in the five nightly steps and the one inpullrequest.yml. The browser-connector job's actual red state is a separate, intermittent[vitest-pool]: Worker exited unexpectedlyafter all 13 tests pass (green on Sep 9 and 10, red on Sep 8 and 11); that test file already carries EPIPE hardening for it and this PR does not change it.Verification (local,
wrangler devper file)chat-recovery5/5 failed,reattach-budgetfailedtest:e2e)work_budget_exceededcase failedtest:e2e)fiber-evictionfailed on the restartoxfmt --check .andoxlintclean.