feat(direct-mode): context awareness — seed from target field + clipboard bridge (RFC 0015) - #52
Merged
Merged
Conversation
…oard bridge (RFC 0015) Implements the frontend half of the RFC 0015 context-awareness amendment (engine CAPI in DasherCore PR #83; design in Dasher-Windows #50): - TargetContextReader: reads the focused control's text + caret via UI Automation TextPattern (modern apps; UTF-16 caret units) with a WM_GETTEXT/EM_GETSEL fallback for classic EDIT/RichEdit. Background thread with a 300 ms hard budget - an unresponsive provider never stalls a mode switch; failures return null and degrade to session-only context. - Seeding: on keyboard-mode entry and on target-window change, the field's text seeds the engine via dasher_seed_buffer (caret converted from UTF-16 with dasher_byte_offset_from_utf16). Target switches RE-READ the new field (better than v5's reset-to-empty); read failure seeds empty (v5 parity). 150 ms debounce for focus churn. - Clipboard bridge on the mini-bar: Copy (engine buffer to system clipboard), and Cut/Paste/Select-All as Ctrl+X/V/A chords injected into the target - they act on the TARGET's selection, which only the target can do (why v5 did them frontend-side). Submodule pinned to DasherCore feat/context-capi pending merge/tag of #83. Signed-off-by: will wade <willwade@gmail.com>
Pins #81 (OutlineWidth 0->1) at the Windows test level: fresh user dir yields LP_OUTLINE_WIDTH=1 and opcode-3 outline rects in the draw stream. Confirms beta guidance: wiping %APPDATA%/Dasher applies the new default (a persisted 0 from earlier builds still masks it - documented, no migration by design for the beta). Signed-off-by: will wade <willwade@gmail.com>
User report: seeding never fired on Notepad; occasionally worked on caret moves. Two causes: 1. TryReadTextPattern used GetFocusedElement() (the globally focused element) — at the moments we read, Dasher itself was often the foreground (mode entry, or the brief window during focus transitions), so UIA returned one of OUR controls with no TextPattern, the read failed, and we silently seeded empty. Now uses ElementFromHandle(targetHwnd): always the target window, never whoever happens to have focus. 2. Mode-entry seeding was premature — the user just clicked Dasher's button, hasn't focused the target yet, and any read at that point is wrong. Removed; the Deactivated handler (which fires the instant the user clicks into the target, with the live caret) is the sole and correct trigger, now also firing on refocus of the same target (caret may have moved). Also adds a ThemeDiagnosticTests frame-dump tool for palette investigations (fill/outline/text colour distribution under European/Asian). Signed-off-by: will wade <willwade@gmail.com>
- Keyboard-mode minimum is now 300x250 (half the normal 600x500), switched per-mode on entry/exit so normal mode keeps its full minimum - TargetContextReader now logs every step (UIA ElementFromHandle, TextPattern probe, Win32 fallback) to %APPDATA%/Dasher/keyboard_debug.log — the seeding path is remote-debugged by reading that file after a test session. Failed reads show exactly which mechanism ran and why it bailed. Signed-off-by: will wade <willwade@gmail.com>
1. Caret offset: CompareEndpoints returns a SIGNED relative comparison (caller vs target endpoint), not an absolute offset — the raw value clamped to 0, anchoring every caret at the document start. Now negated: caret = -result. 2. UIA element resolution: three-step strategy — the globally focused element IF its process matches the target (most precise: actual caret), then the target's focused child HWND, then the top-level, then a FindFirst descendants search for TextPattern. Never reads text outside the target's process. 3. Cross-target security (UIA): every element's process ID is verified against the target's before reading. 4. Cross-target security (Win32): the system-wide GetGUIThreadInfo(0) fallback removed entirely — only the target-thread-scoped focus is used, with a process-ID ownership check before WM_GETTEXT. 5. Top-level misses focused controls: descendants search via FindFirst(TextPattern condition) when the top-level and focused-child elements lack TextPattern. Also: EnsureTargetForeground now re-seeds when it detects a NEW target during typing (the user's first keystroke in a different app is the strongest signal they've arrived at their real target). Signed-off-by: will wade <willwade@gmail.com>
Greptile P1 on #52: FindFirst(TreeScope_Descendants) can reach out-of-process hosted text elements (browser render processes). The descendant's CurrentProcessId is now verified against the target's before its TextPattern enters the engine. Signed-off-by: will wade <willwade@gmail.com>
… doesn't fire in keyboard mode Root cause of context seeding never triggering: the WS_EX_NOACTIVATE style that keeps Dasher from stealing focus also suppresses Avalonia's Deactivated event, so the seed had no trigger. v5 solved this with a WinEventHook (DasherWindow.cpp HandleWinEvent) — this implements the same: SetWinEventHook(EVENT_SYSTEM_FOREGROUND) installed on keyboard-mode entry, removed on exit. Signed-off-by: will wade <willwade@gmail.com>
The hook fired but changedTarget was false when _lastTargetWindow was already set from a prior assignment (ApplyPaneLayout or an earlier Deactivated), suppressing the seed. Now every foreground event in keyboard mode triggers a seed attempt — a same-window re-read is correct (user may have moved to a different field within the same window) and the debounce prevents rapid-fire. Signed-off-by: will wade <willwade@gmail.com>
…cus hook Three Greptile P1s on #52: 1. Caret offset: CompareEndpoints magnitude semantics vary by provider. Replaced with a truncated-range measurement: clone the document range, truncate END to the selection START, GetText — the length IS the caret offset, independent of provider quirks. 2. Cross-window boundary: the process-only check accepted any window in the same process. Now walks the UIA control-view tree from the focused element upward, checking the native window handle matches the target HWND; a different window in the same process is rejected. 3. Same-window field changes: added an EVENT_OBJECT_FOCUS WinEventHook alongside the foreground hook — fires when focus moves between controls within the same window (browser tabs, multi-pane apps), triggering a re-seed with the new field's context. Signed-off-by: will wade <willwade@gmail.com>
Greptile P1 on #52: the OBJID_WINDOW filter rejected EVENT_OBJECT_FOCUS events for non-windowed controls (browser inputs, WPF, Office, Electron editable elements), so same-window field changes never re-seeded. The filter now only applies to EVENT_SYSTEM_FOREGROUND; focus events keep the containing window's hwnd, which is all the seeding path needs. Signed-off-by: will wade <willwade@gmail.com>
Context CAPI (dasher_set_offset, dasher_seed_buffer, UTF-8 converters + clamp hardening) is now tagged. Moves the submodule off the feat/context-capi branch onto the release tag. Signed-off-by: will wade <willwade@gmail.com>
Greptile P1 on #52: EVENT_OBJECT_FOCUS can fire with a child-control HWND (classic EDIT in a dialog, MDI children); storing that in _lastTargetWindow gave SetForegroundWindow a non-top-level handle, breaking target restoration and sending input to the wrong window. The callback now roots the HWND via GetAncestor(GA_ROOT) before storing/comparing, so focus on a child of the current target is treated as a same-window field change and EnsureTargetForeground always gets a top-level handle. Signed-off-by: will wade <willwade@gmail.com>
Greptile P1 on #52: browsers and Electron expose TextPattern from render processes whose PID differs from the window's, so the PID equality gates (focused-element check + descendant lookup) rejected the target's own controls and the Win32 fallback couldn't read them either — seeding null and clearing the buffer. Containment is now proven purely by the control-view tree walk to targetHwnd, which renderer elements satisfy and foreign elements do not. Signed-off-by: will wade <willwade@gmail.com>
Greptile P1 on #52: a GUI thread can own several top-level windows; GetGUIThreadInfo returned the thread's focus regardless of which sibling window held it, and the process-ownership gates accepted it. GetFocusedChildInThread now roots the focused control via GetAncestor(GA_ROOT) and rejects anything not rooted at the requested top-level window — fixing both the UIA focused-child fallback and the Win32 edit fallback in one place. Signed-off-by: will wade <willwade@gmail.com>
…KeyboardTargetTracker The review findings on #52 had converged on one invariant — an HWND/element belongs to the target iff GetAncestor(GA_ROOT) lands on the tracked root — but it was improvised per-site (PID gates, thread-ownership checks, first-non-zero-handle comparisons), each with a legitimate exception found by the next review pass. Restructure: - TargetWindowIdentity: the ONE definition of ownership (RootOf/IsOwnedBy). Fixes the child-window containment failure: a target-owned child HWND hosting TextPattern descendants (browser render-widget hosts, MDI children) is now accepted; sibling top-levels of the same process/thread are still rejected. - KeyboardTargetTracker: owns the WinEventHooks (foreground + focus), the rooted target HWND, and foreground restoration (AttachThreadInput escalation) — extracted from MainWindow, which now only decides what to seed. - TargetContextReader: every acquisition path (focused element, thread-focus child, descendant search) routes through the identity class; redundant PID gates removed. - Regression tests with real Win32 windows: child belongs to root, sibling top-level rejected, tracker roots child handles, own window ignored. Signed-off-by: will wade <willwade@gmail.com>
Contributor
Author
|
Structural response to the review series (9c70336): the eight P1s converged on a single invariant — a handle/element belongs to the target iff GA_ROOT lands on the tracked root — so it now exists in exactly one place instead of being re-derived per site:
|
Greptile P1 on #52: EVENT_OBJECT_FOCUS is system-wide and background apps emit it for internal control focus changes; the callback replaced the tracked target on every rooted focus event, letting a background app's context enter predictions and later receiving activated input. Focus events now only count when their rooted window is the current foreground window — the assignment happens after the guard. Target switches remain covered by EVENT_SYSTEM_FOREGROUND, whose hwnd is foreground by definition. Signed-off-by: will wade <willwade@gmail.com>
Greptile P1 on #52: hook callbacks are delivered asynchronously, so the live foreground check discarded valid same-window field transitions whenever the foreground flickered between event and delivery. The guard is now two-tier: focus events for the already-tracked target are accepted without a foreground query (they cannot hijack anything — the window is already the target), while focus events promoting a NEW target still require the foreground root to match. Signed-off-by: will wade <willwade@gmail.com>
Greptile P1 on #52: the caret measurement always truncated through the selection START, so a non-collapsed selection anchored predictions at the selection beginning regardless of where the caret actually was. Now prefers IUIAutomationTextPattern2.GetCaretRange — the only API reporting the caret's active end during a selection — and falls back to the provider-quirk-proof truncated-range method using the selection END for non-collapsed selections (the active end for forward selections: shift+Right, Ctrl+Shift+End, left-to-right mouse drag); collapsed selections remain endpoint-agnostic. Signed-off-by: will wade <willwade@gmail.com>
…uard Two Greptile P1s on #52: 1. Backward selections: TextPattern1 cannot report selection direction (the documented reason GetCaretRange exists). New fallback tier resolves the exact caret from the blinking system caret — GUITHREADINFO.hwndCaret (verified target-owned) + GetCaretPos + ClientToScreen + RangeFromPoint — before the selection-END heuristic. The residual case (no TextPattern2 + hidden caret) is genuinely undetectable via TextPattern1. 2. Stale same-target focus: the seed path now verifies at SEED time — after the debounce delay, immediately before the read — that the tracked target is still the foreground window (TargetIsForegroundOrUnknown). A queued focus event whose target was superseded mid-debounce is dropped; the new foreground's own event chain seeds. This keeps the delayed-event acceptance in the hook while closing the leak at the point of use. Signed-off-by: will wade <willwade@gmail.com>
Greptile P1 on #52: GetCaretPos reports the CALLING thread's caret — this reader runs on a background thread, so it always failed and fell through to the selection heuristic. The same GetGUIThreadInfo call already provides rcCaret (client coords of the target thread's caret); a point inside that rect goes through ClientToScreen → RangeFromPoint as before. GetCaretPos removed. Signed-off-by: will wade <willwade@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the frontend half of the RFC 0015 context-awareness amendment for Windows (governance PR #37; research in #50; engine CAPI in DasherCore PR #83 — submodule pinned to that branch pending merge/tag).
Dasher 5's "direct mode knows the context" was really a session shadow buffer — it never read the target field. This PR restores that parity and exceeds it: predictions now continue from text already in the target field, which v5 never could.
Context seeding (RFC 0015 tiers 1–3)
TargetContextReader(Services/TargetContextReader.cs): reads the focused control's text + caret via UI AutomationTextPattern(modern apps: WPF, UWP, browsers, Office, Electron) with aWM_GETTEXT/EM_GETSELfallback for classic EDIT/RichEdit controls (Notepad, dialogs). All reads run on a background thread with a 300 ms hard budget — an unresponsive provider can never stall a mode switch; failures returnnull.dasher_byte_offset_from_utf16CAPI (the shared, tested converter from DasherCore #83), so CJK/accented text anchors correctlyKbLogtraces each seedClipboard bridge (mini-bar)
Four new buttons on the keyboard-mode mini-bar:
Ctrl+X/V/Achords injected into the target viaSendInput; they act on the target's selection, which only the target can do — the same reason v5 implemented them frontend-side rather than in control modeVerification
Depends on
dasher_seed_buffer,dasher_set_offset, unit converters) — submodule pinned tofeat/context-capi; re-pin to the tagged release before mergeType of change
Definition of Done
Greptile Summary
The PR adds target-field context seeding and clipboard controls for direct keyboard mode, with centralized target-window tracking and ownership checks.
Confidence Score: 5/5
The PR appears safe to merge.
No blocking failure remains.
Important Files Changed
Sequence Diagram
Reviews (19): Last reviewed commit: "fix(keyboard): caret rect from GUITHREAD..." | Re-trigger Greptile