Skip to content

fix(xterm): remove scrollback decorations by identity - #13178

Merged
nwparker merged 5 commits into
stablyai:mainfrom
bbingz:fix/10879-marker-dispose-order
Sep 8, 2026
Merged

nwparker merged 5 commits into
stablyai:mainfrom
bbingz:fix/10879-marker-dispose-order

Conversation

@bbingz

@bbingz bbingz commented Aug 8, 2026 •

Copy link
Copy Markdown
Contributor

ELI5

Heavy output while terminal search highlights are active repeatedly scans highlights already removed from scrollback. Removing 1,000 highlights requires over 800,000 key reads. Remove decorations by identity so the terminal does constant lookup work per removal, while keeping the same highlights and scrollback.

What Changed

Replace the existing SortedList keyed-delete retry patch with an identity-to-index map. Unique entries store a scalar index; index arrays are allocated only for duplicate identities. Preserve the sorted array, live-key queries, deferred insertion/compaction, duplicate entries, and snapshot iteration. Use a set of deleted indices for linear compaction. Clear/rebuild the index at the existing array lifecycle boundaries.

Regenerate xterm 6.1.0-beta.303 bundles and sourcemaps through the pinned source-patch workflow and update the lockfile hash. No dependency version change or unrelated source-patch change.

Why

The issue still reproduces on current main. The previous version of this PR reordered Marker.dispose notifications, but Buffer.addMarker already subtracts the trim amount before disposal. Ordinary one-line trims still collapse evicted keys to -1. This replacement leaves the public marker event contract unchanged and handles mutated keys directly.

Identity indexing also subsumes the previous missed-delete retry fix. Search limits, retained output, marker disposal timing, and sorted overview-ruler iteration remain unchanged. The index uses O(n) additional storage bounded by entries already retained in the list; it is cleared on reset and rebuilt after compaction.

Linked Issue

Fixes #10879

Visual Proof

Background Electron / Playwright CDP validation on local macOS, using an isolated profile and a folder workspace. App identity was verified against this worktree. Screenshots below show successive states on the final implementation, not a timing comparison against old code.

  1. Opened the actual find bar and matched 1,000 PTY-produced lines: 1,001 decorations including the active match.
  2. Kept the find bar open while a real PTY command produced 6,000 more lines: all 1,001 original decorations removed exactly once, zero remaining index entries, and 5,069 retained lines (5,000 configured scrollback + 69 visible rows).
  3. Reused search for 100 retained matches; next/previous navigation and highlights worked.
  4. Closed search: zero decorations. Typed echo UI_OK_10879 through the terminal input and observed the exact output.

Highlights before trim

highlights-before-trim.png

After trim, with find bar still open

after-trim.png

Search reused successfully

search-reused.png

Search closed; keyboard input/output verified

closed-search-input-ok.png

Testing

  • Automated tests added and tested red/green against the shipped runtime.
  • 152 tests passed across 12 files: new eviction contracts, existing search leak/long-line tests, the xterm-addon.boundary-containment command, and patch-generator contracts.
  • pnpm tc:web.
  • Changed test passes oxlint and oxfmt; git diff --check excluding generated patch files (unified-diff blank context lines intentionally contain a space).
  • ORCA_BACKGROUND_LAUNCH=1 node config/scripts/regenerate-xterm-patches.mjs --check --work-dir=/tmp/orca-xterm-10879 reproduces all shipped patches and lockfile hashes.
  • Frozen-lockfile install succeeds.
  • Actual rendered terminal/find-bar flow on background macOS Electron via CDP; screenshots inspected.
  • Full physical platform/network and native-focus matrix; CI covers broader checks.

Deterministic lookup evidence (250 / 500 / 1,000 disposed entries): baseline 52,789 / 209,788 / 836,287 key reads; replacement 0 / 0 / 0. Real search eviction previously used 838,041 key reads and now satisfies the linear budget. New tests also cover duplicate identity removal, repeated deletes, live-key queries after shifts, snapshot order, reinsertion, clear, and 2,000 seeded mixed operations.

Final scalar-index benchmark (plain Node/V8, 1,000 entries, median of 100 samples after warmup on local macOS): dispose-then-delete plus compaction 1.374 ms → 0.069 ms; normal delete plus compaction 0.055 ms → 0.062 ms; insertion approximately 0.041 ms → 0.067 ms. Whole-terminal timing depends on renderer and other work; the deterministic key-read counts are the performance oracle.

Forced-GC memory measurement across 200 lists, each containing the same 1,000 shared decoration objects (isolates collection storage): original collection approximately 8.5 KB/list; first array-index candidate 93.5 KB/list; final scalar-index candidate 37.5 KB/list. Incremental overhead is therefore approximately 29 KB per 1,000 decorations, down from 85 KB (66% reduction). Clearing drops storage to roughly 0.6 KB per empty final list, with no per-entry retention. n is decoration entries, not scrollback lines. Duplicate identities retain their supported semantics through lazy index arrays and a new partial-compaction/reinsertion test.

Review

Readiness checklist: PASS for this diff. No proven P0, P1, or P2 findings; counsel skipped under the checklist's zero-candidate exception.

Area Evidence / disposition
SSH / remote / network Client-side decoration removal only. No transport, reconnect, host execution, backpressure, or liveness changes.
Crash / cast / retry / growth Production adds no casts, non-null assertions, retries, timers, or external awaits. Missing/repeated deletes return false. New map/set are bounded by list entries; clear and compaction remove stale indices.
Security / supply chain Only SortedList source changes; generated output verified from pinned upstream. No new dependencies, versions, network sinks, shell commands, install hooks, or secrets.
Performance / resources Quadratic key scan removed; measured operation counts and timings above. Existing deferred cleanup scheduling retained. Additional bounded identity index is the implementation cost.
Functional correctness Real Terminal + SearchAddon regression and existing renderer-cell leak oracles pass. Sorted iteration, marker event contract, output and search remain intact. Provider-neutral.
Backcompat / data No persisted state, schemas, migrations, serialization, RPC, wire, or mobile-facing contract touched. Paired web uses the same client-side package without an exchanged-contract change.
Cross-platform / remoting Pure JavaScript collection change, with no OS, path, shell, Git, PTY or local-host assumptions. Local/daemon/SSH/WSL/relay execution ownership unaffected; git worktrees and folder workspaces unaffected.
Organization Extends the existing source patch and places regression beside existing terminal-search tests. No parallel runtime subsystem.

Reliability invariant: terminal-decoration.identity-removal — removing a decoration must notify consumers exactly once even after its marker key changes, retain live decorations in buffer-line order, and leave output intact. Failure source: #10879 and the existing search-highlight leak regression. Oracle: shipped-runtime removal counts, empty/retained decoration iteration, search reuse, output assertions, and deterministic lookup budget. Related existing gate: xterm-addon.boundary-containment (executed). Dedicated eviction performance coverage is in ordinary Vitest CI; a dedicated manifest gate is not added in this change.

Residual validation gaps: physical Windows/Linux/WSL/SSH/relay/mobile, native-focus transitions, and real-network fault recovery were not run. Local macOS has actual Electron screenshot and terminal/input evidence in addition to happy-dom contracts. The first CDP command immediately after focus lost its initial character and the shell rejected it; that attempt was not counted as a pass. After input settled, a complete keyboard-typed command and exact shell output passed. This initial automation anomaly was not diagnosed as a production regression, and native-focus transitions remain unproved. Those implementation paths are unchanged. No negative user-facing behavior trade-off was introduced or observed; this is evidence, not an all-platform guarantee.

Agent skill upstream boundary

  • Not applicable; no upstream skill source copied.

Checklist

  • Focused source change, regenerated artifacts, and regression tests.
  • Root cause and before/after evidence documented.
  • Visual proof N/A explained.
  • Self-reviewed correctness, security, performance, compatibility, and cleanup.
  • Cross-platform, SSH, mobile, and folder-workspace impact considered.
  • Relevant local checks pass; full build and broader matrix remain for CI.

@coderabbitai

coderabbitai Bot commented Aug 8, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: 2c6c16d6-dc43-42a0-9c1f-3e2c4e405bf1

📥 Commits

Reviewing files that changed from the base of the PR and between cbc7bb4 and 7467a83.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (3)
  • config/patches/@xterm__xterm@6.1.0-beta.303.patch
  • config/patches/xterm-src/@xterm__xterm@6.1.0-beta.303.src.patch
  • src/renderer/src/components/terminal-scrollback-decoration-eviction.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

SortedList now tracks value identities with a Map and deleted indices with a Set. Deletion supports duplicate values and values with mutated sort keys. Identity indices are rebuilt after replacement, clearing, and deferred compaction. New tests cover direct SortedList behavior and terminal scrollback decoration eviction, including marker disposal, retained output, bounded lookup work, and highlight recreation.

Merge Risk: ⚪ Minimal · up to 7467a

The change removes quadratic search-decoration cleanup with broad regression coverage and no established behavioral risk. The branch conflicts with main must still be resolved before merging.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 1 files. (1 skipped: 1 … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: removing xterm scrollback decorations by identity.
Description check ✅ Passed The description is comprehensive. It covers the problem, implementation, rationale, linked issue, visual evidence, testing, review risks, and checklist status. Some template items, such as a separate …
Linked Issues check ✅ Passed The PR satisfies issue #10879. It replaces quadratic key-based deletion with identity-based lookup, supports duplicate entries and key mutation, preserves sorted iteration and cleanup behavior, and ad…
Out of Scope Changes check ✅ Passed The reviewed changes are limited to the SortedList patch and focused terminal scrollback-decoration eviction tests. Generated xterm artifacts and lockfile updates are consistent with the stated source…
Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 1 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)
config/patches/xterm-src/@xterm__xterm@6.1.0-beta.287.src.patch (3)

20-20: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Keep native commits visible in screen reader mode.

Line 20 returns before the existing !screenReaderMode guard. The adjacent contract requires insertText to bubble in screen reader mode for character announcement. Do not consume composition input on this path when screen reader mode is enabled.

Proposed fix
-    if (ev.data && ev.inputType === 'insertText' && this._compositionHelper?.handleCompositionInput(ev.data, true)) return true;
+    if (ev.data && ev.inputType === 'insertText' && !this.optionsService.rawOptions.screenReaderMode && this._compositionHelper?.handleCompositionInput(ev.data, true)) return true;

32-34: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Return the non-composing default from the mock.

CoreBrowserTerminal.input now calls this method. This mock reports isComposing === false, but it throws instead of allowing normal input to continue. Return false.

Proposed fix
   public handleCompositionInput(data: string, nativeCommit: boolean): boolean {
-    throw new Error('Method not implemented.');
+    return false;
   }

295-295: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Replace the raw NUL byte with a TypeScript escape.

The delimiter is a raw NUL byte. Git can classify files containing NUL bytes as binary, which breaks reliable patch regeneration and application. Use the textual \0 escape instead.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2c60be8c-b6ea-454e-be46-5f8e0834f826

📥 Commits

Reviewing files that changed from the base of the PR and between 25a8c51 and ac790a5ac8dd3bba7809dac24bbad6781ce373ef.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (3)
  • config/patches/@xterm__xterm@6.1.0-beta.287.patch
  • config/patches/xterm-src/@xterm__xterm@6.1.0-beta.287.src.patch
  • src/renderer/src/components/terminal-marker-dispose-key-order.test.ts

@bbingz

bbingz commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Backlog note (cleanup pass): auto-rebase onto current main hit conflicts. Branch left open; needs manual conflict resolution or a fresh PR before it can land. Not abandoned — just conflicted.

@bbingz
bbingz force-pushed the fix/10879-marker-dispose-order branch from ac790a5 to 511e383 Compare August 10, 2026 03:41
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

Scrollback trim under search highlights was O(k²) because dispose set
marker.line to -1 before onDispose, collapsing SortedList keys. Fire
listeners first so delete still sees the real line, then clear the line.

Fixes stablyai#10879
@bbingz
bbingz force-pushed the fix/10879-marker-dispose-order branch from 511e383 to decd687 Compare August 18, 2026 05:38
@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@nwparker nwparker removed the P1 label Aug 19, 2026
Supersede the old beta.287 marker-event reorder with the tested beta.303 SortedList fix. Preserve the existing PR history while retaining current main patches and regenerated artifacts.
@nwparker nwparker self-assigned this Sep 7, 2026
@nwparker nwparker changed the title fix(xterm): fire Marker dispose before clearing line fix(xterm): remove scrollback decorations by identity Sep 7, 2026
@greptile-apps

greptile-apps Bot commented Sep 7, 2026

Copy link
Copy Markdown

Too many files changed for review (3000 files, 100 file limit).

nwparker added a commit that referenced this pull request Sep 8, 2026
`pr-code-change-scope.mjs` read stdin with `readFileSync(0, 'utf8')`, a single
read of fd 0. Once the writer outgrows the 64 KB pipe buffer that read returns
early or throws EAGAIN, the script exits 0 having emitted no `name=value` pairs,
and `tee -a "$GITHUB_OUTPUT"` records nothing -- so every lane the classifier
gates is silently skipped rather than failing loudly.

A PR opened long ago carries a stale `pull_request.base.sha`, so the gate's
merge-base diff spans the whole base branch. PR #13178 diffed 13,294 files
(773 KB) against a base 1,592 commits behind main and lost typecheck, test,
static analysis, xterm patch sync, package and e2e to this.

Stream stdin instead, matching how the sibling `pr-e2e-source-routing.mjs`
already reads the same list in the same workflow.
@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@nwparker
nwparker merged commit 5bd0247 into stablyai:main Sep 8, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: scrollback trim is O(k²) when evicting decorated lines (search highlights)

4 participants