Skip to content

test(e2e): verify Claude is prefilled with issue URL on start - #9980

Merged
AmethystLiang merged 2 commits into
mainfrom
bug-after-posting-an-issue-start-loads-claude-bu
Jul 22, 2026
Merged

AmethystLiang merged 2 commits into
mainfrom
bug-after-posting-an-issue-start-loads-claude-bu

Conversation

@AmethystLiang

Copy link
Copy Markdown
Contributor

Summary

Adds E2E test to verify that when a user creates a new GitHub issue in Orca and starts a workspace from it, the Claude agent is launched with the issue URL prefilled via the --prefill flag.

This test validates the fix for the regression where newly created issues were not properly passing their context (URL) to Claude on start.

Testing

  • Added high-quality E2E test that validates the complete workflow:
    1. Create a new GitHub issue via Orca UI
    2. Click "Start workspace from issue"
    3. Verify Claude process receives --prefill and --dangerously-skip-permissions flags with the issue URL
    4. Verify the command is properly formatted (regression check for shell quoting)

Test uses mocked gh and claude CLIs to simulate GitHub interaction and capture command arguments. Cross-platform paths handled for Windows and Unix-like systems.

Security Audit

No security concerns. Test file only; uses mocked CLIs and a disposable E2E repository. Fake CLI scripts are isolated to a temporary directory and cleaned up after the test suite.

Notes

Test covers:

  • Cross-platform CLI setup (.cmd batch scripts on Windows, executable shell scripts on Unix)
  • GitHub issue creation flow with mocked responses
  • Terminal content verification via Playwright's poll utility with appropriate timeouts
  • Proper fixture composition with custom launchEnv to inject fake CLI directory into PATH

Regression test for #6613: when starting a workspace from a newly
created GitHub issue, ensure the issue URL is passed to Claude via
`--prefill` and `--dangerously-skip-permissions` flags. This prevents
context loss after issue creation.
@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 5ca2c5ae-7c5f-442b-90e0-0371405e36df

📥 Commits

Reviewing files that changed from the base of the PR and between 20edaad and 847f42d.

📒 Files selected for processing (1)
  • tests/e2e/github-created-issue-start-prefill.spec.ts

📝 Walkthrough

Walkthrough

Adds a Playwright end-to-end test with fake platform-specific gh and claude executables. The test configures a disposable repository, creates a GitHub issue through the UI, starts a workspace from it, and verifies terminal output contains Claude’s expected flags and exact issue URL without malformed quoting.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning It covers summary, testing, security, and notes, but is missing the required Screenshots and AI Review Report sections. Add a Screenshots section with 'No visual change' if applicable, and include an AI Review Report covering risks, checks, and cross-platform validation.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately describes the main regression test added in the PR.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
tests/e2e/github-created-issue-start-prefill.spec.ts (2)

164-164: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Unnecessary RegExp for substring matching.

Playwright's getByRole name option already does case-insensitive substring matching for plain strings by default; wrapping ISSUE_TITLE in RegExp is redundant and is what triggers the static-analysis ReDoS warning. Passing the string directly avoids the warning without changing matching behavior.

-  await expect(orcaPage.getByRole('heading', { name: new RegExp(ISSUE_TITLE) })).toBeVisible({
+  await expect(orcaPage.getByRole('heading', { name: ISSUE_TITLE })).toBeVisible({

Source: Linters/SAST tools


170-180: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Redundant terminal read after the poll succeeds.

The polled content at line 171 is discarded, then getTerminalContent is called again at line 176 for the remaining assertions — an extra IPC round-trip and a small window where content could theoretically differ between the two reads. Capture the polled value directly instead.

♻️ Proposed fix
-  await expect
-    .poll(() => getTerminalContent(orcaPage, 12_000), {
-      timeout: 30_000,
-      message: 'Claude prefill command did not reach the active terminal buffer'
-    })
-    .toContain('--prefill')
-  const terminalText = await getTerminalContent(orcaPage, 12_000)
+  let terminalText = ''
+  await expect
+    .poll(
+      async () => {
+        terminalText = await getTerminalContent(orcaPage, 12_000)
+        return terminalText
+      },
+      {
+        timeout: 30_000,
+        message: 'Claude prefill command did not reach the active terminal buffer'
+      }
+    )
+    .toContain('--prefill')

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: c3cb8b53-d533-41b3-a7ca-8b0cba271eb6

📥 Commits

Reviewing files that changed from the base of the PR and between 0121f57 and 20edaad.

📒 Files selected for processing (1)
  • tests/e2e/github-created-issue-start-prefill.spec.ts

- Reorder mock API handlers to ensure `/labels` and `/assignees` paths match before the specific issue endpoint
- Replace regex heading matcher with exact string for more reliable assertions
- Refactor terminal content polling to capture text once and reuse in subsequent assertions
@AmethystLiang
AmethystLiang merged commit 9bdb820 into main Jul 22, 2026
1 check passed
enrick-dev pushed a commit to enrick-dev/orca that referenced this pull request Oct 15, 2026
…ai#9980)

* test(e2e): verify Claude is prefilled with issue URL on start

Regression test for stablyai#6613: when starting a workspace from a newly
created GitHub issue, ensure the issue URL is passed to Claude via
`--prefill` and `--dangerously-skip-permissions` flags. This prevents
context loss after issue creation.

* test(e2e): fix GitHub-issue-start prefill test flakiness

- Reorder mock API handlers to ensure `/labels` and `/assignees` paths match before the specific issue endpoint
- Replace regex heading matcher with exact string for more reliable assertions
- Refactor terminal content polling to capture text once and reuse in subsequent assertions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant