Skip to content

feat: replace playwright area snapshot with cdp ax tree#24

Open
cryptotavares wants to merge 5 commits intomainfrom
cryptotavares/replace-playwright-area-snapshot-with-cdp-ax-tree
Open

feat: replace playwright area snapshot with cdp ax tree#24
cryptotavares wants to merge 5 commits intomainfrom
cryptotavares/replace-playwright-area-snapshot-with-cdp-ax-tree

Conversation

@cryptotavares
Copy link
Copy Markdown
Collaborator

@cryptotavares cryptotavares commented May 5, 2026

What

Replaces Playwright's ariaSnapshot() with raw CDP accessibility APIs for building the accessibility tree that agents consume, and extracts a shared CDP session lifecycle helper.

Why

The discovery layer relied on Playwright's YAML-based ariaSnapshot() API, which has no concept of backend DOM node IDs. Generic-role elements (popover menu items, custom list rows, unnamed wrappers) could never be mapped to stable Playwright selectors — they were silently dropped from snapshots, leaving agents blind to actionable UI that doesn't use standard ARIA roles.

How

CDP accessibility integrationcollectTrimmedA11ySnapshot now opens a CDP session and queries the browser's accessibility tree directly:

Scope CDP method Fallback
Full page Accessibility.getFullAXTree
Scoped (root selector) Accessibility.queryAXTree Accessibility.getPartialAXTree

Every AX node now carries its backendDOMNodeId, enabling DOM-level selector resolution for nodes that don't map to a clean ARIA role selector.

Selector resolution strategy — Nodes are resolved to Playwright selectors via a tiered approach:

  1. Included ARIA roles (button, link, heading, etc.) → cheap role=…[name="…"] selector, no CDP round-trip
  2. Generic/non-standard rolesDOM.resolveNodeRuntime.callFunctionOn to synthesize the most stable CSS selector available:
    • Unique data-testid[data-testid="…"]
    • Unique id#…
    • Ancestor path → div > div:nth-of-type(1) > span
    • Text fallback → text="…" (last resort, with debug warning)

Named generic nodes — Generic-role nodes that carry a non-empty accessible name (e.g. popover action items) are now included in snapshots. Unnamed generic wrappers are still excluded but their children are traversed.

CDP session lifecycle — A new withCdpSession(page, fn) helper centralises session creation → use → guaranteed detach, shared with the clipboard tool.

Test coverage — Comprehensive tests for CDP session lifecycle (detach on success/failure), scoped snapshots (queryAXTree and getPartialAXTree fallback), generic node inclusion/exclusion, DOM selector resolution, ignored node handling, and ancestor-path selectors. Coverage thresholds updated accordingly.

@cryptotavares cryptotavares requested a review from a team as a code owner May 5, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant