Skip to content

feat(computer-use): Linux support for KDE and GNOME - #207

Draft
coreybain wants to merge 26 commits into
mainfrom
feat/computer-use-linux
Draft

coreybain wants to merge 26 commits into
mainfrom
feat/computer-use-linux

Conversation

@coreybain

@coreybain coreybain commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Draft, not ready to merge. This parks the Linux (KDE/GNOME) half of Computer Use so it isn't lost. The macOS half shipped in b6e9ce3.

What's here

  • P8 Linux port (12 commits): Linux Cua admission and host, availability/JPEG preview contracts, server hooks for Linux backends, KWin plugin and D-Bus plumbing, session-bus probes and evdev input, app launch, provisioning, wl-clipboard, KWin plugin provisioning, AT-SPI perception.
  • Fixes from the part-1 review (12 commits): input can't cross a desktop replacement, previews don't freeze on a swap, threads seeded before a swap see the new desktop, and the availability revision rule: the newest read decides, publishing never reads, and a swap cancels the pending window republish.
  • Linux CI (2 commits): build the KWin plugin per distribution, and check the Linux Cua host and patched driver.

Hyprland is deliberately left out.

Still to do before this can merge

  1. Verify the last fix round. V2-01, V2-02 and V2-03 (stale availability after a failure, older reads overwriting newer ones, an old publication driving a new desktop) were fixed in the last five fix commits. The verification run was stopped before it gave a verdict.
  2. Review the other 8 P8 commits. The review so far covered only the first 4 (host, contracts, server hooks, web). KWin, session bus/evdev, app launch, provisioning, clipboard and AT-SPI have never been reviewed.
  3. Check GNOME. Neither Synara nor this port has a GNOME-specific backend; GNOME relies on the generic Cua/AT-SPI/evdev path, which is untested on a GNOME Wayland session.
  4. Related Mac follow-up d-4: the clients currently advertise Linux support. This PR makes that true once it lands; if it stays parked, hide it instead.

Checks run on this branch

  • Rebased onto main f7384a3; all 26 commits applied without conflicts.
  • tsgo passes for contracts, server, web and desktop.
  • Server Computer tests: 1,281 passed, 32 skipped (84 files).
  • Not run on a real KDE or GNOME session.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

coreybain and others added 26 commits September 27, 2026 07:16
Ports Synara's Linux admission rules, the task-scoped X11 Escape
shortcut and the Linux desktop host (#1294 core hooks). Linux stops
failing closed: the Cua host now receives linuxAdmission, and Computer
use runs on Linux behind PATHWAY_COMPUTER_USE. Host errors keep the
message of plain SurfaceErrors instead of printing [object Object].

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…h app ids

A Linux server may not know its desktop yet at startup, so availability
gains a transient "checking" member. Preview frames carry their image
type (PNG stays the default and the only model-facing format), health can
say a backend is dormant by choice, and a launch can name the app id its
windows report when a wrapper handed the window off.

Ported from Synara pr-1294 (5f3f50397e).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adds the seams the KWin and Hyprland backends plug into: a switchable
backend slot so startup never waits on a wedged session bus, a Linux
backend selection with a strict PATHWAY_COMPUTER_BACKEND override, and
manager support for desktop-gone reselection, passive status reads,
dormant health, luma scroll baselines, per-backend settle policy,
observed-paste clipboard restore, launch app ids, JPEG preview frames and
pane-input marking.

Ported from Synara pr-1294 (5f3f50397e); Synara's tests are the spec.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…views

A checking availability shows as checking rather than as a setup to do,
on web and iOS, and the live preview decodes each frame as the image
type its header names.

Ported from Synara pr-1294 (5f3f50397e).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ports Synara's KWin compositor plugin literally: the plugin, its
dedicated agent seat, the D-Bus interface, the install and uninstall
scripts, the systemd rebuild timer and the fixture tests that compile
production functions standalone. Product names are renamed: Synara to
Pathway, org.synara.ComputerUse to com.spiritdevs.pathway.ComputerUse,
SYNARA_* env vars to PATHWAY_*. The tree lives at native/computer-use-kwin
beside Pathway's other natives.

Ported from Synara pr-823 (8f8d804828).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ports Synara pr-823's D-Bus layer to Effect: variant unwrapping, typed
call/timeout/connection-closed failures, the connection watch that
dbus-next itself lacks, the scoped per-bus session token, and the KWin
session-bus adapter with owner pinning and throttled-auth cooldown.
Connection-level classifiers live in dbusFailures.ts, and the plugin/bus
doubles are Effect-returning.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Port Synara's sessionBusNames, evdevInput, pointerSequencing and
asarUnpackedPath (pr-823). The probes run on a throwaway watched
connection and fail with DbusConnectionClosedError when the bus dies.
Sequencing takes Effect sinks and reads time from the Effect Clock;
releases run uninterruptibly after the body, whatever its exit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Port Synara's appLaunchResolution, desktopAppEnvironment and
computerHealthState (pr-823). Resolution fails through Effect with
AppLaunchResolutionError, a ComputerBackendError; health state is a
factory rather than a class.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ports Synara's KWin provisioning helpers: distribution and toolchain
probes, KWin compatibility, prebuilt verification, the flock-based
provisioning lock, the from-source plugin build, and the pkexec package
installer. Cancellation is fiber interruption throughout; runCancellable
runs its child in a process group through the Effect spawner, and pkexec
is only ever signalled while it is still waiting for its dialog.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Reads, writes and paste-once offers on the seat0 clipboard now run as
Effects. The paste-once offer's end is a Deferred with a bounded watch
fiber. wl-copy is still spawned directly, because the Effect spawner's
scope would take down the fork that serves the selection. The KWin
installer-script tests run in the ported sandbox testkit, on Linux only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Installs the KWin plugin under the user's home, preferring a verified
prebuilt and falling back to a source build under the installer lock.
Cancellation is interruption; every failure is a ComputerBackendError.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ports Synara's AT-SPI helper (Python, literal with pathway renames), the
helper client on the Effect process spawner, tree fusion and the
perception layer the KWin engine reads trees through.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ement

When the 64-entry operation queue was full, replaceDesktop ran the swap
outside the queue and aborted only agent calls. A pane click waiting on the
old desktop's window list then sent its input to the replacement desktop.

The swap now enters the queue as a barrier the admission limit cannot
refuse, and replacement aborts in-flight pane input as well as agent calls.
Only a closed queue lets the swap run directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…lients

Each still-frame publisher numbered its frames from 1, and the manager sent
that number on unchanged. After a desktop replacement, web and iOS clients
that kept their frame socket open still held the old stream's higher
sequence, so they dropped every new frame as stale.

The manager now stamps the outgoing sequence from its own counter, so it
keeps rising for the computer stream across backend instances. Neither
client's sequence gate or the frame protocol changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ktop

A thread seeded while Linux backend selection was still running kept
reporting "checking" after the selected backend arrived. The swap's health
and capability events only republish the cached thread snapshot, so the
thread showed the placeholder until it asked again. A later replacement left
threads showing the gone desktop in the same way.

Replacing the desktop now invalidates the physical snapshot, and ignores an
in-flight read of the previous occupant. It then pushes the new occupant's
passive availability to every seeded thread without establishing or
provisioning it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The barrier test forked the barrier lazily, so it asked for admission only
after the held operation had been released and the queue had drained.
Admitting barriers under the ordinary 64-entry limit still passed.

The barrier now starts immediately and must be waiting, not refused, while
all 64 entries are still held.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… its probe

After a desktop swap, the manager pushes a passive probe of the new
desktop to every thread. If a request had already engaged the manager
during detection, a later request could establish the new desktop and
succeed while the probe was still running. The probe then landed on an
empty snapshot, pushed its older "unavailable" verdict, and every later
health event republished it.

Every establishing availability read now goes through one helper. The
first one to succeed after a replacement becomes the threads' verdict and
is pushed to them, and the probe still in flight is dropped. With no real
use, the probe behaves as before and never starts the desktop.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After a desktop swap, the manager cached availability through three
separate writers, each with its own guard. A cached read failure outlived
a later success, so threads kept showing "unavailable". A slow passive or
establishing read could overwrite a newer verdict for the same desktop.
A push that was still running when the desktop was replaced again read
the backend and started the new, unused desktop.

Each availability read now takes a revision when it starts. The cache
accepts a result only if it is newer than the last one it accepted. A
replacement starts the cache at a fresh revision, so reads that began
before it land nowhere. A newer success clears an older failure. A
failed window or screen read is kept as the thread's error but never
hides the availability. Publication only copies the cache, never reads
the backend. A push about one desktop stops once that desktop is
replaced. The one-time replacement flag is gone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cross-replacement test only counted establishing reads on the next
desktop. Publication no longer reads the backend, so that count stayed at
zero even with the generation check removed. The test now also checks that
no push about the first desktop lands after the swap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…op swaps

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… reads the new desktop

The debounced republish after a window change does a full physical read, so a
timer armed for the old desktop could fire after a swap and run establishing
reads on the new one. The swap now retires the old occupant inside its barrier:
it bumps the generation, resets the cache and cancels the pending timer. A
timer that fires anyway checks the generation it was armed in and drops itself.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…republish

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Mirrors Synara's KWin prebuild matrix on the Linux fleet runner: one
digest-pinned container per distribution, KWin read off the image, and
the .so plus a JSON sidecar uploaded. The KWin fixture tests run as
their own job. x64 only, with no manifest assembly yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Mirrors Synara's Cua Linux workflow: Linux host and protocol tests,
the pinned source built with both Pathway patches, and its browser and
cursor tests. The Electron X11 smoke is not ported yet.
docs/internals/computer-use-ci.md records the three workflows, their
gaps from Synara, and what the Hyprland lane will need.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL labels Sep 26, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant