feat: replace playwright area snapshot with cdp ax tree#24
Open
cryptotavares wants to merge 5 commits intomainfrom
Open
feat: replace playwright area snapshot with cdp ax tree#24cryptotavares wants to merge 5 commits intomainfrom
cryptotavares wants to merge 5 commits intomainfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces Playwright's
ariaSnapshot()with raw CDP accessibility APIs for building the accessibility tree that agents consume, and extracts a shared CDP session lifecycle helper.Why
The discovery layer relied on Playwright's YAML-based
ariaSnapshot()API, which has no concept of backend DOM node IDs. Generic-role elements (popover menu items, custom list rows, unnamed wrappers) could never be mapped to stable Playwright selectors — they were silently dropped from snapshots, leaving agents blind to actionable UI that doesn't use standard ARIA roles.How
CDP accessibility integration —
collectTrimmedA11ySnapshotnow opens a CDP session and queries the browser's accessibility tree directly:Accessibility.getFullAXTreeAccessibility.queryAXTreeAccessibility.getPartialAXTreeEvery AX node now carries its
backendDOMNodeId, enabling DOM-level selector resolution for nodes that don't map to a clean ARIA role selector.Selector resolution strategy — Nodes are resolved to Playwright selectors via a tiered approach:
role=…[name="…"]selector, no CDP round-tripDOM.resolveNode→Runtime.callFunctionOnto synthesize the most stable CSS selector available:data-testid→[data-testid="…"]id→#…div > div:nth-of-type(1) > spantext="…"(last resort, with debug warning)Named generic nodes — Generic-role nodes that carry a non-empty accessible name (e.g. popover action items) are now included in snapshots. Unnamed generic wrappers are still excluded but their children are traversed.
CDP session lifecycle — A new
withCdpSession(page, fn)helper centralises session creation → use → guaranteed detach, shared with the clipboard tool.Test coverage — Comprehensive tests for CDP session lifecycle (detach on success/failure), scoped snapshots (
queryAXTreeandgetPartialAXTreefallback), generic node inclusion/exclusion, DOM selector resolution, ignored node handling, and ancestor-path selectors. Coverage thresholds updated accordingly.