Skip to content

visionOS: close the probe-first agent loop and document human gates #264

Open
Open
visionOS: close the probe-first agent loop and document human gates#264
Labels
enhancementNew feature or requestinfraRepo / tooling / releaseplatform:visionosApple Vision Pro / visionOSpriority:p2Normal planned work with meaningful impactstatus:needs-reconcileIssue facts or acceptance criteria may lag the current repository
Milestone
Validation Automation

Description

@jlipworth
jlipworth
opened on Aug 12, 2026

Parent umbrella: #182

Goal

Define and close the maximum safe visionOS agent loop supported by current Xcode 27 tooling, with explicit human/headset gates for unsupported interaction.

Current foundation

  • Passive scripts/agent-sim-run.sh artifact harness.
  • Credential-free launch-fixture-home-passive launch state.
  • Build/run/log orchestration through official Xcode MCP.
  • Playback/download probes and playback-frame capture.
  • Official Xcode Device Interaction currently rejects visionOS; the legacy standalone-Simulator click path is not supported by Device Hub.

Acceptance

  • Credential-free passive fixture launch produces complete evidence under the one-simulator lease.
  • Playback/download correctness uses deterministic app-side probes where UI driving is unnecessary.
  • The maintained skill clearly blocks free-form/legacy clicking and identifies gaze, pinch-drag, immersive UI, and physical-device checks requiring human/headset confirmation.
  • Closure does not claim semantic visionOS UI automation until Apple exposes and a bounded run proves it.

Activity

  1. jlipworth
    added
    enhancementNew feature or request
    infraRepo / tooling / release
    on Aug 12, 2026

  2. jlipworth commented on Aug 12, 2026

    @jlipworth
    jlipworth
    on Aug 12, 2026
    OwnerAuthor

    Credential-free visionOS passive closure was re-proven on Xcode/visionOS 27 with scripts/agent-sim-run.sh launch-fixture-home-passive --duration 3.

    The run passed and retained start/end screenshots, MP4, app/simulator/build logs, and machine-readable run.json; the end screenshot visibly rendered the synthetic Home catalog. Local evidence: artifacts/agent-sim-runs/20260812T193041Z-launch-fixture-home-passive. Simulator was shut down afterward.

    The maintained skill continues to block legacy/free-form clicking. Official Device Interaction remains iOS-only, so semantic visionOS gaze/pinch/immersive validation is still an explicit human/headset gate rather than an unproven automation claim.

  3. jlipworth commented on Aug 12, 2026

    @jlipworth
    jlipworth
    on Aug 12, 2026
    OwnerAuthor

    Integrated locally into main through f24412c0. The credential-free passive visionOS fixture produces complete evidence, and the maintained simulator skill documents the Xcode 27 Device Interaction limitation plus gaze/pinch/immersive/headset gates without endorsing free-form clicking.

    Publication status: local main only; not yet pushed to origin/main. Leaving open until publication and maximum-supported-scope review.

  4. jlipworth
    added this to the Validation Automation milestone on Aug 22, 2026
  5. jlipworth
    added
    priority:p2Normal planned work with meaningful impact
    status:needs-reconcileIssue facts or acceptance criteria may lag the current repository
    platform:visionosApple Vision Pro / visionOS
    on Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestinfraRepo / tooling / releaseplatform:visionosApple Vision Pro / visionOSpriority:p2Normal planned work with meaningful impactstatus:needs-reconcileIssue facts or acceptance criteria may lag the current repository

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions