Skip to content

Run the picked Claude model for AI text generation - #37367

Merged
brennanb2025 merged 5 commits into
mainfrom
brennanb2025/claude-textgen-haiku
Nov 2, 2026
Merged

brennanb2025 merged 5 commits into
mainfrom
brennanb2025/claude-textgen-haiku

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​208 $\color{#cf222e}{\Huge{\mathbf{−}}}$​23 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​185
Prod 5 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​128 $\color{#cf222e}{\Huge{\mathbf{−}}}$​20 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​108

Problem

Picking Claude "Haiku" for AI text generation (commit message, PR details, branch name, chat name, Jira summary) ran Claude Sonnet instead. You picked the fast, cheap model and got a slower, pricier one, with nothing in Orca saying so. On main you reach it through the setup guide's AI author card or --model haiku in an action's Arguments.

The cause: Orca kept Claude from editing files during text generation by starting it in plan mode (--permission-mode plan). In plan mode the Claude CLI swaps the haiku alias for Sonnet (and opusplan for Opus). Plan mode also made Claude write plan files into the Claude config folder whenever a prompt asked for an edit.

Change

Claude text generation no longer uses plan mode, and the model you pick is the model that runs. What Claude may do while writing the text:

  • Read and search files (Read, Grep, Glob) anywhere your account can read, inside or outside the checkout. A saved custom prompt that says "read commit-style.txt for the prefix" or names a file elsewhere on disk keeps working.
  • No commands and no edits. Orca gives Claude no shell, write or edit tool, and it no longer leaves plan files behind. (Company MCP tools on older CLIs are the one exception; see below.)
  • Your own permission rules still apply. A path you deny or mark "ask" in your Claude settings stays refused, because text generation can't stop to ask you.
  • Older Claude CLIs keep working. Versions before 2.0.43 don't know the flags this uses; Orca retries once in the older syntax with the same file-reading policy, never plan mode.

Model ids stay as the provider/account aliases, so Bedrock and Vertex keep resolving them as before. Tracked in STA-13412.

How it works

  • Current CLIs (2.0.43 and later) get --tools Read,Grep,Glob --allowedTools Read,Grep,Glob --permission-mode dontAsk. --tools limits the built-in tools to the three file tools, --allowedTools lets them read outside the checkout, and dontAsk refuses anything that would need approval. An explicit mode also overrides a defaultMode: "plan" in the user's settings.
  • Older CLIs refuse dontAsk with argument 'dontAsk' is invalid. On exactly that completed refusal Orca retries once with --disallowedTools <every other built-in those versions have> --allowedTools Read(//**),Grep(//**),Glob(//**) --permission-mode default. //** means any absolute path; those versions convert Windows paths to /c/... form before matching, so it covers every drive. The retry uses path rules rather than bare tool names because on those versions a bare grant would also override a user's "ask" rule for a specific path.
  • On company setups with a managed MCP config, Orca already retries once without --strict-mcp-config because the CLI refuses that flag there. That retry now also denies all MCP tools (mcp__*) on current CLIs, so it has the same tools as the normal attempt, which loads no MCP servers.
  • Both retries are one-way and happen at most once each (at most three runs), each rebuilt from the user's original settings, so prompts, model, command and Arguments are unchanged. Timeouts, cancellation, provider errors and other failures never retry.
  • The tool set is one constant, CLAUDE_TEXT_GENERATION_TOOLS in src/shared/claude-background-run-isolation.ts; the catalog, the grants and the older-CLI deny list are all derived from it. The older-CLI list is fixed: it is every built-in tool name in the published CLIs that need the older syntax (1.0.69–2.0.42), and only those see it.
  • Paired runtimes and SSH hosts build this argv from their own version; the retry reacts to the CLI that actually ran. Nothing changes on the wire, and nothing is saved.

Differences from the common pattern

  • The common pattern gives commit, PR, branch and title generation no tools at all; Orca keeps file reads: intended. Saved custom prompts in Orca can ask Claude to read files and must keep working (product decision). The read access uses the common pattern's own read-only mechanism (--tools plus a matching --allowedTools under dontAsk), without its shell commands.
  • A one-time retry in older-CLI syntax: intended. The common pattern passes --tools/dontAsk unconditionally and so needs Claude CLI 2.0.43 or later; Orca has no minimum Claude CLI version, and Haiku worked on older CLIs before this change. Older CLIs now also read files outside the checkout, so every supported version follows the same policy.
  • MCP tools denied on the managed-MCP retry: intended. Handling that case with dontAsk alone lets a pre-approved MCP tool that changes things run (reproduced with a local MCP server); plan mode on main refused it.
  • Older CLIs (before 2.0.43) on a company setup with a managed MCP config keep their existing MCP permissions: intended compatibility exception. Those versions can't deny every MCP server at once (their rule matcher has no mcp__* wildcard), and their plan mode never refused pre-approved MCP tools either, so a company-approved MCP tool can still run there, as it could on main. Their built-in tools are still only Read, Grep and Glob.
  • No --disable-slash-commands: intended. The flag only exists from Claude CLI 2.0.60, while the current-syntax attempt serves 2.0.43 and later; adding it would need another older-CLI fallback.
  • No --no-session-persistence: temporary, tracked in STA-13454. Each text-generation run is still saved as a Claude session, exactly as on main.
  • Model ids are provider/account aliases, not catalog ids: intended (Bedrock/Vertex resolve aliases to their own ids; no per-release id table).
  • Plain text output, not a JSON schema: intended (Orca parses text output the same way across all agents).

Behavior changes

  • Saved custom prompts that ask Claude to run a shell command (for example "use git log") get no shell tool; on main plan mode let read-only commands run. Claude replies without running it. File reads keep working.
  • opusplan, listed only when it is the user's own CLI model, now runs Sonnet. That is what it does outside plan mode.
  • A Bedrock or Vertex user who picked Haiku without Haiku access silently got Sonnet. They now get the provider's error through the existing failure message.
  • On the managed-MCP retry, a pre-approved read-only company MCP tool no longer runs during text generation (plan mode allowed it); the normal attempt never had MCP tools.

Evidence

Real Claude CLI, each run in its own throwaway home, signed in through the team load balancer, with defaultMode: "plan", a Read deny rule and a Read ask rule for two outside files in the settings. The model was read from Claude's own transcript.

  • Before (main's argv, CLI 2.1.310): haiku ran claude-sonnet-5-5; plan files were written.
  • Current CLI 2.1.310, final argv, Haiku: claude-haiku-5-5 ran in dontAsk; tools exactly Glob, Grep, Read; the inside file, an outside file and an outside Grep all worked; the denied and "ask" files were refused.
  • Published CLI 2.0.30, final older-syntax argv, Haiku: the retry's argv ran Haiku in default mode; tools exactly Glob, Grep, Read; inside read, outside read and outside Grep worked; the denied and "ask" files were refused.
  • Every older release: an offline check (no sign-in, nothing sent to a provider) of every published release from 1.0.69, the oldest that accepts Orca's other flags, to 2.0.42 shows each one refuses the new flags with the exact text the retry matches, accepts the older-syntax argv including the path grants, and is left with exactly Glob, Grep and Read; 2.0.43 accepts the new flags. The //** root and Windows path handling were checked in the source of all 90 of those releases.
  • Managed-MCP retry (CLI 2.1.310, local stdio MCP server with a read-only and a file-writing tool, both pre-approved in settings): main's plan mode ran the read-only tool and refused the writer; dontAsk without the MCP deny ran the writer (file written); with mcp__* denied no MCP tool was available and nothing was written.
  • Not exercised: a real enterprise managed-MCP machine (emulated with --mcp-config), Bedrock/Vertex (no such account), Windows/WSL/SSH hosts (argv passes through unchanged; no new characters that need quoting).
  • Tests: argv contract for the current, managed-MCP and older-CLI forms (including the read grants on every rebuilt attempt); retry triggers and their composition; request-level managed-MCP and old-CLI-plus-managed-MCP runs.

Claude text generation used plan mode to stay read-only, and in plan mode
the Claude CLI runs Sonnet for the haiku alias. Turn tools off and set
the dontAsk permission mode instead, so every alias runs the picked model
while text generation still cannot edit files or run tools. Tracked in
STA-13412.
Text generation now gets Read, Grep and Glob under dontAsk instead of no
tools, so saved prompts that read repo files keep working. Claude CLIs
older than 2.0.43 refuse dontAsk; Orca retries once with the older
syntax (default mode plus a fixed deny list of the other built-ins those
versions have), never plan mode. The managed-MCP retry denies MCP tools,
matching the strict attempt, which loads none.
dontAsk refuses Read outside the working directory unless the tools are
also granted, so current CLIs now get --allowedTools Read,Grep,Glob
beside --tools. The older-CLI retry grants the same tools as root-path
rules (Read(//**) and so on); a bare grant there would override a user's
path-specific ask rule.
@brennanb2025
brennanb2025 marked this pull request as ready for review November 2, 2026 09:59
@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 10 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2941af58-9f00-4ed4-b631-05227e75c311
Commits

Reviewing files that changed from the base of the PR and between 9a51438 and 4d87b3c.

Files selected for processing: 2
  • src/main/text-generation/jira-issue-summary-text-generation.test.ts
  • src/shared/claude-background-run-isolation.test.ts
Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a3650fee-e229-4500-8b97-dc8a955cd998
Commits

Reviewing files that changed from the base of the PR and between 255401c and 9a51438.

Files selected for processing: 9
  • src/main/text-generation/agent-generation-request.ts
  • src/main/text-generation/jira-issue-summary-text-generation.test.ts
  • src/shared/claude-background-run-isolation.test.ts
  • src/shared/claude-background-run-isolation.ts
  • src/shared/commit-message-agent-spec.test.ts
  • src/shared/commit-message-agent-spec.ts
  • src/shared/commit-message-agent-specs-primary.ts
  • src/shared/commit-message-plan.test.ts
  • src/shared/commit-message-plan.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 1 remain after this review.

Walkthrough

Claude text generation now builds tool arguments for modern and legacy CLI policies. Completed refusals can trigger retries that change strict-MCP or legacy-tool-policy settings. Generation requests replan using the full attempt configuration. Tests cover argument construction, refusal detection, retry order, and a remote CLI fallback.

Merge Risk: Moderate · previous review 9a514

On older Claude CLIs that hit both the managed-MCP refusal and the dontAsk refusal, the final retry can leave managed MCP tools available during text generation. This undercuts the stated goal of generation that cannot run commands or edit files. Restrict the legacy retry to attempts that still have strict MCP enabled, or deny mcp__* in the legacy arguments, before merging.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title "Run the picked Claude model for AI text generation" matches the main change. The PR stops Claude text generation from using plan mode, which made the CLI replace the picked Haiku alias with…
Description check Passed The description follows the repository template. It has Problem, Change, How it works, Differences, Behavior changes, and Evidence sections, which cover the template's What Changed and Why content. It…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "Run the picked Claude model for AI text generation" matches the main change. The PR stops Claude text generation from using plan mode, which made the CLI replace the picked Haiku alias with Sonnet. The title names the picked model and the text-generation scope, so a reviewer can see the primary change. It does not mention the file-read tool policy or the older-CLI retry, but those are secondary details.

Full details: Description check

Explanation

The description follows the repository template. It has Problem, Change, How it works, Differences, Behavior changes, and Evidence sections, which cover the template's What Changed and Why content. It gives detailed Risk-relevant behavior changes and a linked issue (STA-13412), and it reports testing evidence. Several template sections are missing or incomplete: the ELI5 section, an explicit Linked Issue heading with a 'Fixes #' reference (it uses a Linear link instead), the Visual Proof section (required for behavior changes, but no before/after media or N/A reason is given), the Testing checklist boxes, the Risk Level (the 'Level' field is unfilled), the AI Disclosure section, the Review section, and the Agent skill upstream boundary checkbox. The Checklist is also unchecked. These gaps are non-critical overall, so the description is mostly complete.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 6 functions across 9 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
Generate docstrings
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@brennanb2025

Copy link
Copy Markdown
Contributor Author

Review loop status

Problem. Picking Claude "Haiku" for AI text generation (commit message, PR details, branch name, chat name, Jira summary) ran Claude Sonnet. Orca started Claude in plan mode to keep it from editing files, and in plan mode the Claude CLI swaps haiku for Sonnet (and opusplan for Opus). Plan mode also left plan files in the Claude config folder.

What users get now. The model you pick is the model that runs. Claude can read and search files (inside or outside the checkout, subject to your own deny/ask rules) but can't run commands or edit anything, so saved custom prompts that read a file keep working. Older Claude CLIs (before 2.0.43) keep working through one retry in their older syntax, never plan mode. On company setups with a managed MCP config, current CLIs deny MCP tools on that retry.

How review went. Full path, 3 review rounds plus a CI repair, two design-panel rulings.

  • Round 1 found that removing every tool (the first version) broke saved custom prompts that read files, and that older CLIs reject the new flags. Brennan ruled: Claude may read files but not run commands, built the common way. Implemented with the common pattern's read-only tool mechanism.
  • The panel added a one-time older-CLI retry that stays out of plan mode, and an MCP deny on the managed-MCP retry. On older CLIs, reads are granted with path rules so a user's own "ask" rule still applies.
  • Round 2 found reads outside the checkout were still refused. The fix adopts the common pattern's matching allow list.
  • The final reference comparison asked for one disclosure in the description: older CLIs keep their own managed-MCP permissions. Added and labelled intended.
  • CI-1 fixed a type error in a test this PR added (test-only).

Deferred / not in this PR.

  • Each text-generation run is still saved as a Claude session that shows in Session History, the same as on main: temporary, tracked in STA-13454.
  • On main, text generation sends --effort for Sonnet and Opus without checking the CLI version, so CLIs older than 2.1.38 fail with those defaults. This is a separate bug for Brennan's list, not fixed here.

Verified.

  • Real Claude CLI 2.1.310 with the exact production argv at the final head: a saved prompt reading an in-repo file plus a file outside the checkout gave the right commit message on haiku, sonnet and opus, and each transcript shows the picked model.
  • Default PR details (valid JSON), branch name and chat name ran Haiku.
  • A diff asking Claude to write a file created nothing.
  • A prompt asking it to run git log did not run anything and leaked no tool-call text, but Claude appended a sentence saying it couldn't run the command. That follows from the no-commands ruling.
  • Older-CLI argv was checked live on published CLI 2.0.30 and offline on all 90 releases from 1.0.69 to 2.0.42.
  • The managed-MCP deny was checked with a local MCP server.
  • Unit and request-level tests cover the argv, the retry triggers and their composition.

Not verified. A real enterprise managed-MCP machine, Bedrock/Vertex, and live Windows/WSL/SSH hosts (argv passes through unchanged).

QA evidence: real Claude CLI 2.1.310 run through the team load balancer in throwaway homes, against the production planCommitMessageGeneration argv and prompt builders at 9a51438 (commit, PR, branch and chat flows; haiku/sonnet/opus). Logs are in the loop's QA folder (pr-37367/qa-report.md, /tmp/pr-37367-qa). Commit 4d87b3c since then changes test files only.

For Brennan's review

  • src/shared/claude-background-run-isolation.ts, claudeTextGenerationToolArgs: the tool policy for current and older CLIs, including the MCP deny.
  • src/shared/claude-background-run-isolation.ts, runClaudeGenerationAttempts: the two one-way retries (managed MCP, older CLI).
  • The "Behavior changes" section of the description: saved prompts asking for shell commands no longer run them.

@brennanb2025
brennanb2025 merged commit 210c624 into main Nov 2, 2026
54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant