Skip to content

feat(simulation): expose and record real mouse input - #47194

Merged
kitlangton merged 1 commit into
v2from
simulation-mouse
Sep 4, 2026
Merged

feat(simulation): expose and record real mouse input#47194
kitlangton merged 1 commit into
v2from
simulation-mouse

Conversation

@kitlangton

Copy link
Copy Markdown
Contributor

Why

Simulation can click an element, but cannot move a pointer through real hover/leave states or hold a button while dragging. Recorded terminal output also contains no mouse coordinates, so Drive cannot show viewers where those interactions happened.

What Changes

Add optional ui.mouse and ui.recording.pointer capabilities without changing existing click behavior or the terminal-v1 recording format.

yield* ui.mouse({ action: "move", x: 20, y: 8 })
yield* ui.mouse({ action: "down", x: 20, y: 8, button: "left" })
yield* ui.mouse({ action: "move", x: 30, y: 8 })
yield* ui.mouse({ action: "up", x: 30, y: 8, button: "left" })
yield* ui.mouse({ action: "scroll", x: 30, y: 8, direction: "down" })
  • Coordinates are absolute zero-based cells within the current viewport. Moves, buttons, modifiers, and scroll dispatch through the existing OpenTUI mouse harness.
  • Existing target-relative clicks record their resolved absolute position.
  • A lazy *.pointers.jsonl sidecar timestamps input on the same monotonic clock as terminal output. Older terminal recording readers remain compatible.
  • Finishing or failing the recording joins both output streams, preserving the original error and cached finish result.

Demo

spring-smooth.mp4

Real production TUI controlled by Drive: native action hover and leave, target-relative click, down/up theme toggles, and resize. The pointer is composited during export by the companion Drive change; interpolation sends no additional input. Captured with this implementation before its clean rebase (52974a6057), rendered with Drive 961e75b; no model requests or playback acceleration. This PR adds simulation capabilities, not a production TUI layout change.

Scope

Canonical simulation protocol, real renderer input dispatch, and portable mouse recording evidence. Cursor styling and spring interpolation remain Drive-owned. Existing operations continue to work with clients that do not request the new capabilities.

Verification

cd packages/simulation
bun run test
bun typecheck
cd ../client
bun run generate
  • 46 simulation tests pass, including real hover/drag/right-button/scroll and recording sidecar/error lifecycle cases.
  • Client generation leaves no generated diff. The normal pre-push hook passes all 33 package typechecks.
  • Real isolated Drive demo validates hover/leave highlighting, click results, and resizing.
  • Independent review reproduced three companion recording/lifecycle defects; all original reproductions pass after fixes, with 50 tests and four targeted cases in the follow-up review.

@kitlangton
kitlangton enabled auto-merge (squash) September 4, 2026 02:07
@kitlangton
kitlangton merged commit 19833ad into v2 Sep 4, 2026
10 checks passed
@kitlangton
kitlangton deleted the simulation-mouse branch September 4, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant