Skip to content

fix(agent): let Wave read plan replies itself - #2835

Merged
dylan-savage merged 3 commits into
feat/rocket-agent-browserfrom
fix/rocket-agent-plan-json
Oct 28, 2026
Merged

dylan-savage merged 3 commits into
feat/rocket-agent-browserfrom
fix/rocket-agent-plan-json

Conversation

@krishhgg

@krishhgg krishhgg commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Base: feat/rocket-agent-browser at fda7c67 (PR #2718).

What broke

Each wave, Wave asks the engine for its plan with expectJson: true. The engine then parses the reply strictly. It accepts one JSON value and nothing after it, retries the call up to 3 times, and then answers with an LLM error: ValueError: Failed to get valid JSON response after 3 attempts. createChatModel throws on that answer, and the loop ends the turn.

GPT-5.5 usually writes its plan and then more fenced JSON blocks: a copy of the plan, a "waiting for results" plan, or plans for later waves. The engine refuses all of these. In the baseline on 369f051, all 10 GPT-5.5 turns across 5 trials ended this way, and none wrote a pipeline. Claude hit the same error once in 439 calls, when it wrote its plan as markdown.

Wave never saw those replies. Its own reader, parseReply, takes the first fenced block and already read 12 of the 13 GPT replies we captured. Its retry note covers the 13th.

The fix

  • buildWaveQuestion (wave/plan.ts) sends the plan question with expectJson: false. The engine returns the reply as text, and parseReply reads it. An unreadable reply goes back once with Wave's retry note, the same path as any other unusable reply.
  • Without expectJson, the engine no longer adds its own "JSON Response Format" instruction. Wave's Response Format already asks for fenced JSON, and parseReply reads the first block, so the prompt is unchanged.

Other effects on the engine side, for review:

  • A bad reply now costs one Wave retry instead of up to 3 full-size engine calls and an error.
  • The engine takes its normal chat path for the plan call. That path streams, and sends the model's reasoning lines as thinking events. The best-effort question already went this way.

Since Dylan's review

  • The tests are one 9-line loop test. plan.test.ts, the behindEngine helper and the pasted GPT-5.5 replies are gone, since 7 of the 8 old tests passed without the fix.
  • The WAVE_RESPONSE_FORMAT JSDoc and the comment above expectJson: false are one line each.
  • The one-block line this PR had added to the prompt is gone. Replies still had extra blocks with it, and parseReply reads the first block either way.

Tests

  • wave/loop.test.ts, one test. The model's reply is the plan's block and then a block with done: true, as GPT-5.5 writes it. The plan question goes out with expectJson: false, the plan's call runs, and the run goes on to the next wave instead of finishing on the second block.
    • With expectJson: true put back, the test fails on the flag.
    • With parseReply reading the last block instead of the first, it fails because the run ends on the second block's answer.
  • No test of parseReply's reply shapes, because this PR does not change parseReply.

Test output

On fda7c67 with this PR:

  • ./builder shared:test: 227 tests, 227 pass, 0 fail. 1 is new. The tip alone has 226.
  • npx tsc --noEmit -p tsconfig.json in apps/shared, apps/rocket-ui and packages/shell: no errors. It needs ./builder client-typescript:build first.
  • ./builder rocket-ui:build: passes.
  • prettier --check and eslint on the three changed files: clean.
  • gitleaks on the PR's commits: no findings.

Logs are kept locally (round 3, json-fix review logs of 2026-10-28).

GPT-5.5 before and after

These runs were on the earlier bases and were not repeated on c560a80. They are the same 5 baseline trials: build-01-chat-claude t1 to t3 and build-02-summarize-webhook t1 to t2, through the stand-in with llm_openai:gpt-5-5. The base changed between the runs, from 369f051 to 8bd0122. Between them, 66646e8 changed the core prompt and the guides, and 8bd0122 replaced the pipeline toolkits with one rocketride_client tool.

Before, 369f051 After, 8bd0122 with this fix
Trials finished 0 of 5 5 of 5
Turns that ended on the engine's JSON error 10 of 10 0 of 5
Model errors at least 8 0
Waves per turn 1 to 4 7 to 35
.pipe files written 0 5, all validate
Scenario checks 0 of 5 pass 4 of 5 pass
  • 72 of the 88 plan replies after the fix still had more than one JSON block. The engine's parseJson refuses every reply of that shape in the raw capture, so with expectJson on, these turns would have hit the same error. parseReply read all 88, and none needed a retry.
  • A prompt line asking for exactly one block did not stop the extra blocks in these runs, so the PR leaves the prompt alone.
  • The one failed check set is build-01 t3. Its pipeline leaves out profile on the Anthropic node and relies on the node's default. The engine validated it.

The run records are kept locally: the round 3 baseline runs openai (before) and openai-after-json-fix (after), compared in the baseline README.

Browser check

Earlier run, on c560a80 with the first version of this PR: 3 of 4 checks pass. One turn, 94 seconds. pipelines/verify-chat.pipe was written, read back through the engine, and validated with no errors. The reply ends "Checked: validate OK".

The page sent 11 plan questions, all with expectJson: false and cachePrefix: true.

The reply-recorded check fails, and the cause is in the base. The chat could not save the session: the console shows session save failed Error: Unknown subcommand: fs_put_json. Commit 9bd5d93, from feat/appdev-cloud, made fsWriteJson send fs_put_json, and only the branch's engine has that command. The journey runs the published engine image. On the bare tip the journey fails the same check with the same console error, and the other 3 pass. There the page sent 15 plan questions, all with expectJson: true.

On 8bd0122, this change passed all 4 checks (2026-10-24). On the final stack after Dylan's review, on an engine built from the branch, it passes all 4.

Evidence kept locally: the build-pipeline runs of 2026-10-25 at 05:30 UTC (this commit) and 05:33 UTC (bare tip).

The stack

Eleven PRs, each on the one below it. Merge top-down: each PR squashes into the branch below it, starting with #2885 and ending with #2835 into feat/rocket-agent-browser. A top-down rehearsal in a scratch clone was clean at every step and ended on #2885's tree. Bottom-up also works if each next branch is first rebased onto the squashed base.

  1. fix(agent): let Wave read plan replies itself #2835 plan replies (this PR)
  2. fix(agent): keep secrets out of the model in the browser agent #2836 secrets
  3. fix(agent): keep a run in its team, start nothing over a timed-out call, and ignore markers of deleted files #2839 team, timeouts and deleted files
  4. fix(agent): give every file a version, so the agent and an editor cannot write over each other #2840 versioned writes
  5. fix(agent): keep one document per open file and pipeline, so splits, closed tabs and saves stay in step #2882 editor documents and the rename fix
  6. feat(agent): ask before the agent changes a file with unsaved changes, and stop the tool clock while the user answers #2883 unsaved-changes dialog and tool clock
  7. fix(agent): check a staged file against the copy the page read, so Commit all meets a teammate's change #2884 manual-mode commit check
  8. feat(agent): end each turn with a result card, let the agent ask the user, and stop only its own runs in Auto #2869 turn result, needs you and run ownership
  9. feat(agent): pause a turn at its step or cost limit, show its estimated cost, and undo its file changes #2873 step and cost limits, cost estimate, Undo
  10. feat(agent): check the browser agent end to end in a real browser, with a verify skill for each layer of the stack #2874 verify skill
  11. feat(agent): replay the browser agent's smoke scenarios in CI, from fresh recordings #2885 eval runner

@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 2
  • develop
  • release/.*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor
🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

@krishhgg
krishhgg force-pushed the fix/rocket-agent-plan-json branch from fd5fcfe to 4e6d3a9 Compare October 27, 2026 05:57
@github-actions github-actions Bot added module:ui Chat UI and Dropper UI tests labels Oct 27, 2026
The plan question goes out without expectJson. The engine's JSON check
takes one value and nothing after it, and after 3 tries it answers with an
error that ends the turn. GPT-5.5 often writes its plan and then more fenced
blocks. parseReply takes the first, and the loop sends an unreadable reply
back once. Response Format gains a line asking for exactly one block.

@dylan-savage dylan-savage left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change itself is fine, but the tests are too big for it.

plan.test.ts is 164 lines for a one-line flag flip, and 7 of its 8 tests pass on the base. They re-test parseReply's first-fence rule, which this PR doesn't change, using about 6 KB of pasted GPT-5.5 replies. Keep one test per behaviour the fix relies on: the plan question goes out without expectJson (this test or the behindEngine loop test, not both), and a later done: true block doesn't finish the run. Cut the fixtures to the fields you assert, and aim for about 40 lines. While you're in there, cut the GPT-5.5 history in the WAVE_RESPONSE_FORMAT JSDoc and the comment above expectJson: false to one line each. The PR body already has it.

One follow-up outside this PR. Plan calls now stream, and a stream that fails after its first token comes back empty with no retry (chat.py:553-576), so the turn ends in error.

One loop test covers what the fix relies on. The plan question goes out
without expectJson, and a done block after the plan does not finish the run.
The parseReply shape tests passed without the fix, so they are gone, with the
fake engine check and the pasted model replies. The comment above expectJson
is one line.

The one-block line in the prompt is gone too. Replies still had extra blocks
with it, and parseReply reads the first block either way.
@krishhgg

Copy link
Copy Markdown
Collaborator Author

Cut to one 9-line loop test. It covers both behaviours the fix relies on: the plan question goes out without expectJson, and a done: true block after the plan doesn't finish the run. It fails with the flag flipped back, and it fails if parseReply takes the last block. plan.test.ts, the behindEngine helper and the pasted replies are gone, and both comments are one line.

I also dropped the one-block line I'd added to the prompt. Replies still had extra blocks with it, and parseReply reads the first block either way. The PR is now +12 −2 across plan.ts and loop.test.ts.

The streaming retry is on our follow-up list.

@dylan-savage dylan-savage left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Thanks for cutting it down.

* fix(agent): keep secrets out of the model in the browser agent

Every call's outcome in the Wave loop passes through redactText: its
result, its error, its prepare error, its refusal and a late result.
File reads and searches show a «secret:<id>» marker for each secret
value, and writes and edits put the value back. The server's guides
under .rocketride/ show as served, and any ${NAME} value counts as a
placeholder, so the guides' examples reach the model whole.

A message that looks like a pasted key never reaches the model, and
its row shows the key as [redacted], live or queued. The title, saved
rows, history, approval cards, messages sent during a run and the
final answer are redacted.

search_files given a file path searches that file.

* fix(agent): hide NAME=value secrets and six more key shapes

- .env, .toml, .ini, .properties and .sh files: the value of each
  secret-named NAME=value shows as a marker, read the same way as a YAML
  field. In an export line each assignment is its own value. It may
  follow spaces after =, as dotenv reads it, and it runs over its words
  up to the next word that sets a name. A word that sets a name right
  after NAME= is read both ways: dotenv reads it as NAME's value, a shell
  as its own assignment. It is hidden when either reading makes it
  secret, and where both do, the assignment keeps its own value. A quote
  left open runs to the end of the line. A TOML string may sit in triple
  quotes, and in a """ string a backslash escapes the next character. A
  .properties value keeps a # that follows a space. In the other files a
  # after = and a space starts a comment, so NAME= # note sets nothing,
  while NAME=#x sets #x. In a shell or .env file, $NAME names another
  variable and shows as written.
- Google (AIza), xAI, Perplexity, Tavily, Pinecone and Firecrawl keys are
  sure shapes, each at its key's full length, so a pasted one is held back.
- Every tool result and error now shows a value the file tools hid as its
  marker, not only a save's note. So do approval cards and the answer,
  whose checks line quotes check_app's summaries. Each toolkit hides those
  values before it cuts a long text, so a cut cannot leave part of one.

* fix(agent): hide a short all-letter value only in its own field

A value the file tools hid showed as its marker wherever it appeared once it
had 8 characters. Dev passwords such as postgres, password and changeme are
also words in node names, guides and answers, so those became markers too.

Now a value shows as its marker outside its own field only when it has 16
characters, or 8 with one that is not a letter. In its own field it is
hidden either way.

---------

Co-authored-by: Krish Garg <krish.garg@rocketride.ai>
@dylan-savage
dylan-savage merged commit 008bdd5 into feat/rocket-agent-browser Oct 28, 2026
4 checks passed
@dylan-savage
dylan-savage deleted the fix/rocket-agent-plan-json branch October 28, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ui Chat UI and Dropper UI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants