Agent troubleshooting: bounded live-playback evidence contract and semantic controls #290

Decision / why this has teeth

Add a bounded, opt-in live-playback troubleshooting contract, extending existing probes and semantic Accessibility—not a general-purpose REST server or an agent-only replacement for the UI.

Recent native Mac playback investigation repeatedly required full Accessibility snapshots, transient menu discovery, and manual chrome revelation just to inspect quality, copy-versus-encode decisions, consent, seek progress, and cleanup. Auto-hiding chrome and stale AX references increase retries. Computer Use remains appropriate for visual inspection, but should not be the sole way to establish these machine-readable facts.

This is a focused follow-up under #182, complementary to #262 (isolated Mac browse fixture), #264 (visionOS probe-first workflow), and #226 (human-facing accessibility). It does not reopen or duplicate their baseline runner work. Related real-playback motivator: #288.

Existing foundation (reuse, do not replace)

• scripts/agent-macos-run.sh / agent-macos-ax-driver.swift: bounded semantic Home-to-detail fixture flow, isolated identity, exact-PID cleanup, JSON artifacts.
• scripts/perf-macos-ax-driver.swift: additional semantic browse/search/artwork automation; performance evidence has its own contract and must remain isolated.
• Labstream/Shared/Debug/Debug{Plex,Jellyfin,Emby}PlaybackProbe.swift: DEBUG-only, explicitly launched, authenticated in-process probes using production playback paths, seek/progress checks, and cleanup.
• DebugPlaybackFrameCapture: decoded frame evidence; transport success alone does not establish correct picture.
• scripts/diagnostics-summarize.py: bounded, privacy-conscious triage/delta artifacts; retain this rather than dumping raw logs into agent context.
• CustomPlayerChrome.swift: native accessible controls exist; current playback changes also add consent identifiers. A complete stable player selector/state contract is not established merely by those two identifiers.

Remaining gap

The Mac fixture runner exposes only passive Home and Home-to-detail scenarios. Existing live playback probes exercise predetermined launch-time sequences, not a unified, versioned contract for observing an already-running playback session and exercising quality/track/consent transitions. They do not replace confirmation that the visible UI is attached to the controller under test.

We need to distinguish requested quality, negotiated video copy/encode/unknown, audio conversion, active rendered format, consent-pending state, actual time progression, and terminal cleanup—without inferring them from a generic “transcoding” label or reconstructing them from repeated screenshots.

Bounded proposal

1. Define the evidence contract first. Versioned allowlisted JSON snapshot/events: build identity, backend enum, opaque session/generation, controller-to-visible-window attachment, playback phase, position/buffer buckets, requested quality, video/audio decisions with provenance, HDR/rendered format where available, consent state, bounded error category, cleanup status. Unknown facts remain explicitly unknown. No raw server responses.
2. Expand named DEBUG probe scenarios, using the same production launcher/controller operations: Original start, seek, track change, capped/Maximum selection, pending consent → decline, and separately authorized consent → approve. Require explicit live-session admission; fixture mode is default. Emit finite pass/fail/blocked artifacts with deadlines and cleanup in cancellation/error paths.
3. Complete semantic player Accessibility for controls and selected/disabled/pending states. Keep labels useful to VoiceOver and identifiers independent of localization/private media names. Add a semantic way to reveal/reach player controls without coordinate clicking; verify behavior when chrome hides, playback buffers, and a menu/consent overlay is active. This portion may ship as ordinary accessibility improvements coordinated with Accessibility follow-ups from Apple UI idiom sweep #226.
4. Evaluate an interactive DEBUG adapter only if named scenarios + AX leave a measured gap. Prefer a local CLI adapter around the same typed contract; choose transport in a small ADR. Do not pre-commit to HTTP, MCP, App Intents, or a shipping control endpoint. Read-only snapshot/export is the first operation; mutation is separately admitted and scoped to the exact playback generation.

Security / release boundaries

• No default listener, LAN exposure, arbitrary URL fetching, shell execution, credential export, or auth bypass. A custom URL scheme is not caller authentication; loopback alone is not authorization.
• DEBUG control code and launch hooks must be absent/inert in Release and PerformanceAudit, verified by existing artifact-contract checks. A future production diagnostics feature requires its own explicit design/review, not accidental promotion of debug hooks.
• Default to isolated app identity and synthetic fixtures. Reusing a signed-in production-identity development build requires explicit user opt-in; never reset its container or Keychain as part of this workflow.
• If IPC is justified: authenticated local peer/capability, short lifetime, restrictive filesystem permissions where applicable, bounded payload/rate limits, exact process/build/session binding, and immediate revocation on stop, logout, backend change, or app exit. Reject stale generations and replayed mutation requests.
• Never auto-approve a video encode because “automation mode” is active. Preserve the real UI approval boundary; an approval scenario must be separately authorized and cannot grant durable future approval.
• No credentials, real hosts/IPs, media titles/paths, raw item/client IDs, or token-bearing URLs in default JSON or public artifacts. Frame capture is separately opt-in and private by default. Diagnostic enablement must be explicit, bounded, and restored.
• Report server-job cleanup as confirmed only with appropriate evidence; a client stop request alone is not proof. Never bypass OS Accessibility trust or locked-session restrictions.

Alternatives / platform guidance

• AX/XCUITest only: lowest additional attack surface and essential for visible UI proof; still constrained by ephemeral chrome and platform automation limits.
• Extend existing launch probes only: preferred initial implementation for deterministic playback facts; does not prove visible menu interaction or hardware presentation.
• App Intents: potentially appropriate for future user-facing actions, not automatically a privileged diagnostics channel. Apple supports explicit authentication policies; any adoption needs a separate supported-platform and authorization review.
• Always-on app API/MCP server: not justified by current evidence; creates unnecessary security, lifecycle, sandbox, and release obligations. A future external MCP wrapper could call the bounded local tooling rather than embedding a server in the app.

Apple references: SwiftUI accessibility identifiers, UI automation with Xcode, App Intent authentication policy.

Acceptance criteria

• Document the contract and transport/no-transport decision, with precise distinction between fixture, live-controller, visible-UI, and hardware evidence.
• One command runs a bounded fixture playback transition scenario and emits schema-validated pass/fail/blocked JSON without screenshots as the state oracle.
• Opt-in live scenarios cover Plex/Jellyfin/Emby Original/copy, seek, quality change, consent decline, separately authorized approval, and cleanup where supported; unsupported cases report blocked, not pass.
• Semantic Mac regression covers hidden chrome, buffering, quality/track selection, and pending consent without coordinates or localized text as the sole selector. Preserve VoiceOver labels, focus, and actions.
• Negative tests reject stale session/generation commands, unauthorized mutation, missing auth, cancellation/timeouts, backend changes, and secret-bearing artifact fields.
• Release/PerformanceAudit checks demonstrate no debug command surface; isolated runner cleanup cannot affect another app identity or process.
• Compare the same troubleshooting scenario before/after: AX/screenshot calls, elapsed time, retries, artifact bytes, and unproven assertions. Claim savings only from measured runs.
• Keep a small actual visual/frame check and physical-device gates for black frames, tint, HDR/audio output, and platform interactions; API success cannot close these.

Investigation-only issue: no implementation, app launch, simulator interaction, or live playback mutation performed for this assessment.