Repository navigation
fix(agent): let Wave read plan replies itself - #2835
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review: 2
Please check the settings in the CodeRabbit UI or the
|
🤖 Internal: Discord sync markerAuto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete. |
fd5fcfe to
4e6d3a9
Compare
The plan question goes out without expectJson. The engine's JSON check takes one value and nothing after it, and after 3 tries it answers with an error that ends the turn. GPT-5.5 often writes its plan and then more fenced blocks. parseReply takes the first, and the loop sends an unreadable reply back once. Response Format gains a line asking for exactly one block.
4e6d3a9 to
48e6647
Compare
dylan-savage
left a comment
There was a problem hiding this comment.
The change itself is fine, but the tests are too big for it.
plan.test.ts is 164 lines for a one-line flag flip, and 7 of its 8 tests pass on the base. They re-test parseReply's first-fence rule, which this PR doesn't change, using about 6 KB of pasted GPT-5.5 replies. Keep one test per behaviour the fix relies on: the plan question goes out without expectJson (this test or the behindEngine loop test, not both), and a later done: true block doesn't finish the run. Cut the fixtures to the fields you assert, and aim for about 40 lines. While you're in there, cut the GPT-5.5 history in the WAVE_RESPONSE_FORMAT JSDoc and the comment above expectJson: false to one line each. The PR body already has it.
One follow-up outside this PR. Plan calls now stream, and a stream that fails after its first token comes back empty with no retry (chat.py:553-576), so the turn ends in error.
One loop test covers what the fix relies on. The plan question goes out without expectJson, and a done block after the plan does not finish the run. The parseReply shape tests passed without the fix, so they are gone, with the fake engine check and the pasted model replies. The comment above expectJson is one line. The one-block line in the prompt is gone too. Replies still had extra blocks with it, and parseReply reads the first block either way.
|
Cut to one 9-line loop test. It covers both behaviours the fix relies on: the plan question goes out without I also dropped the one-block line I'd added to the prompt. Replies still had extra blocks with it, and parseReply reads the first block either way. The PR is now +12 −2 across The streaming retry is on our follow-up list. |
dylan-savage
left a comment
There was a problem hiding this comment.
Approving. Thanks for cutting it down.
* fix(agent): keep secrets out of the model in the browser agent
Every call's outcome in the Wave loop passes through redactText: its
result, its error, its prepare error, its refusal and a late result.
File reads and searches show a «secret:<id>» marker for each secret
value, and writes and edits put the value back. The server's guides
under .rocketride/ show as served, and any ${NAME} value counts as a
placeholder, so the guides' examples reach the model whole.
A message that looks like a pasted key never reaches the model, and
its row shows the key as [redacted], live or queued. The title, saved
rows, history, approval cards, messages sent during a run and the
final answer are redacted.
search_files given a file path searches that file.
* fix(agent): hide NAME=value secrets and six more key shapes
- .env, .toml, .ini, .properties and .sh files: the value of each
secret-named NAME=value shows as a marker, read the same way as a YAML
field. In an export line each assignment is its own value. It may
follow spaces after =, as dotenv reads it, and it runs over its words
up to the next word that sets a name. A word that sets a name right
after NAME= is read both ways: dotenv reads it as NAME's value, a shell
as its own assignment. It is hidden when either reading makes it
secret, and where both do, the assignment keeps its own value. A quote
left open runs to the end of the line. A TOML string may sit in triple
quotes, and in a """ string a backslash escapes the next character. A
.properties value keeps a # that follows a space. In the other files a
# after = and a space starts a comment, so NAME= # note sets nothing,
while NAME=#x sets #x. In a shell or .env file, $NAME names another
variable and shows as written.
- Google (AIza), xAI, Perplexity, Tavily, Pinecone and Firecrawl keys are
sure shapes, each at its key's full length, so a pasted one is held back.
- Every tool result and error now shows a value the file tools hid as its
marker, not only a save's note. So do approval cards and the answer,
whose checks line quotes check_app's summaries. Each toolkit hides those
values before it cuts a long text, so a cut cannot leave part of one.
* fix(agent): hide a short all-letter value only in its own field
A value the file tools hid showed as its marker wherever it appeared once it
had 8 characters. Dev passwords such as postgres, password and changeme are
also words in node names, guides and answers, so those became markers too.
Now a value shows as its marker outside its own field only when it has 16
characters, or 8 with one that is not a letter. In its own field it is
hidden either way.
---------
Co-authored-by: Krish Garg <krish.garg@rocketride.ai>
Base:
feat/rocket-agent-browserat fda7c67 (PR #2718).What broke
Each wave, Wave asks the engine for its plan with
expectJson: true. The engine then parses the reply strictly. It accepts one JSON value and nothing after it, retries the call up to 3 times, and then answers with an LLM error:ValueError: Failed to get valid JSON response after 3 attempts.createChatModelthrows on that answer, and the loop ends the turn.GPT-5.5 usually writes its plan and then more fenced JSON blocks: a copy of the plan, a "waiting for results" plan, or plans for later waves. The engine refuses all of these. In the baseline on 369f051, all 10 GPT-5.5 turns across 5 trials ended this way, and none wrote a pipeline. Claude hit the same error once in 439 calls, when it wrote its plan as markdown.
Wave never saw those replies. Its own reader,
parseReply, takes the first fenced block and already read 12 of the 13 GPT replies we captured. Its retry note covers the 13th.The fix
buildWaveQuestion(wave/plan.ts) sends the plan question withexpectJson: false. The engine returns the reply as text, andparseReplyreads it. An unreadable reply goes back once with Wave's retry note, the same path as any other unusable reply.expectJson, the engine no longer adds its own "JSON Response Format" instruction. Wave's Response Format already asks for fenced JSON, and parseReply reads the first block, so the prompt is unchanged.Other effects on the engine side, for review:
thinkingevents. The best-effort question already went this way.Since Dylan's review
plan.test.ts, thebehindEnginehelper and the pasted GPT-5.5 replies are gone, since 7 of the 8 old tests passed without the fix.WAVE_RESPONSE_FORMATJSDoc and the comment aboveexpectJson: falseare one line each.parseReplyreads the first block either way.Tests
wave/loop.test.ts, one test. The model's reply is the plan's block and then a block withdone: true, as GPT-5.5 writes it. The plan question goes out withexpectJson: false, the plan's call runs, and the run goes on to the next wave instead of finishing on the second block.expectJson: trueput back, the test fails on the flag.parseReplyreading the last block instead of the first, it fails because the run ends on the second block's answer.parseReply's reply shapes, because this PR does not changeparseReply.Test output
On fda7c67 with this PR:
./builder shared:test: 227 tests, 227 pass, 0 fail. 1 is new. The tip alone has 226.npx tsc --noEmit -p tsconfig.jsonin apps/shared, apps/rocket-ui and packages/shell: no errors. It needs./builder client-typescript:buildfirst../builder rocket-ui:build: passes.prettier --checkandeslinton the three changed files: clean.Logs are kept locally (round 3, json-fix review logs of 2026-10-28).
GPT-5.5 before and after
These runs were on the earlier bases and were not repeated on c560a80. They are the same 5 baseline trials: build-01-chat-claude t1 to t3 and build-02-summarize-webhook t1 to t2, through the stand-in with
llm_openai:gpt-5-5. The base changed between the runs, from 369f051 to 8bd0122. Between them, 66646e8 changed the core prompt and the guides, and 8bd0122 replaced the pipeline toolkits with onerocketride_clienttool..pipefiles writtenparseJsonrefuses every reply of that shape in the raw capture, so withexpectJsonon, these turns would have hit the same error.parseReplyread all 88, and none needed a retry.profileon the Anthropic node and relies on the node's default. The engine validated it.The run records are kept locally: the round 3 baseline runs
openai(before) andopenai-after-json-fix(after), compared in the baseline README.Browser check
Earlier run, on c560a80 with the first version of this PR: 3 of 4 checks pass. One turn, 94 seconds.
pipelines/verify-chat.pipewas written, read back through the engine, and validated with no errors. The reply ends "Checked: validate OK".The page sent 11 plan questions, all with
expectJson: falseandcachePrefix: true.The
reply-recordedcheck fails, and the cause is in the base. The chat could not save the session: the console showssession save failed Error: Unknown subcommand: fs_put_json. Commit 9bd5d93, from feat/appdev-cloud, madefsWriteJsonsendfs_put_json, and only the branch's engine has that command. The journey runs the published engine image. On the bare tip the journey fails the same check with the same console error, and the other 3 pass. There the page sent 15 plan questions, all withexpectJson: true.On 8bd0122, this change passed all 4 checks (2026-10-24). On the final stack after Dylan's review, on an engine built from the branch, it passes all 4.
Evidence kept locally: the build-pipeline runs of 2026-10-25 at 05:30 UTC (this commit) and 05:33 UTC (bare tip).
The stack
Eleven PRs, each on the one below it. Merge top-down: each PR squashes into the branch below it, starting with #2885 and ending with #2835 into
feat/rocket-agent-browser. A top-down rehearsal in a scratch clone was clean at every step and ended on #2885's tree. Bottom-up also works if each next branch is first rebased onto the squashed base.