Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/release-cut.yml
Original file line number Diff line number Diff line change
Expand Up @@ -871,6 +871,12 @@ jobs:
node config/scripts/runtime-file-watcher-fault-harness.mjs
ELECTRON_RUN_AS_NODE=1 pnpm exec electron config/scripts/runtime-file-watcher-fault-harness.mjs

- name: Gate SSH relay watcher process isolation
run: |
# Why: the remote native watcher shares a daemon with live PTYs.
# Kill only its child and require both PTY and watch recovery before packaging.
node config/scripts/relay-watcher-fault-harness.mjs

- name: Publish release artifacts (Linux)
if: matrix.platform == 'linux-x64' || matrix.platform == 'linux-arm64'
uses: nick-fields/retry@v4
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/release-mac-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,12 @@ jobs:
node config/scripts/runtime-file-watcher-fault-harness.mjs
ELECTRON_RUN_AS_NODE=1 pnpm exec electron config/scripts/runtime-file-watcher-fault-harness.mjs

- name: Gate SSH relay watcher process isolation
run: |
# Why: the remote native watcher shares a daemon with live PTYs.
# Kill only its child and require both PTY and watch recovery before packaging.
node config/scripts/relay-watcher-fault-harness.mjs

- name: Publish release artifacts (macOS)
uses: nick-fields/retry@v4
with:
Expand Down
89 changes: 79 additions & 10 deletions config/reliability-gates.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -3750,7 +3750,7 @@
},
{
"id": "runtime-files.watcher-process-isolation",
"title": "Runtime file watcher faults stay process-isolated and recover without host starvation",
"title": "Runtime and SSH relay watcher faults stay process-isolated without disrupting host services",
"maturity": "experimental",
"protection": "partial",
"owner": "runtime-platform",
Expand All @@ -3759,7 +3759,8 @@
"headless orca serve",
"paired file explorer",
"runtime editor external-change detection",
"desktop filesystem watcher"
"desktop filesystem watcher",
"SSH relay filesystem watcher and live PTYs"
],
"platforms": [
"macos",
Expand All @@ -3779,19 +3780,22 @@
"remote-runtime",
"ssh"
],
"coverageNotes": "Deterministic tests cover a four-child healthy runtime pool plus at most four bounded quarantine children, one quarantine attempt per watch lifetime, queue-aware cancellation, bounded initial and crash-resubscribe crawls, physical desktop-install abort, stale-generation rejection, per-child stat bounds, final-overflow RPC delivery, and renderer eviction before terminal callbacks can retry. A built-entry macOS harness drives the real supervisor, kills its child with SIGSEGV, then requires automatic resubscription and a later event. Windows keeps the existing fs.watch runtime path; SSH shared-watch setup uses registration-owned cancellation so one caller cannot cancel another caller's watch.",
"coverageNotes": "Deterministic tests cover a four-child healthy runtime pool plus at most four bounded quarantine children, one quarantine attempt per watch lifetime, queue-aware cancellation, bounded initial and crash-resubscribe crawls, physical desktop-install abort, stale-generation rejection, per-child stat bounds, final-overflow RPC delivery, and renderer eviction before terminal callbacks can retry. A built-entry macOS harness drives the real supervisor, kills its child with SIGSEGV, then requires automatic resubscription and a later event. SSH now bundles the same child boundary beside relay.js, requires both artifacts for install completeness, and preserves registration-owned same-root cancellation. A built-relay harness kills that child and requires the original relay PID and existing PTY to survive before overflow and precise events resume.",
"motivatingLinks": [
"https://github.com/stablyai/orca/issues/5308",
"https://github.com/stablyai/orca/issues/7547",
"https://github.com/stablyai/orca/issues/8212"
],
"invariant": "Local runtime roots use at most four shared watcher children while healthy and at most four additional quarantine children after shard failure. Cancelling queued setup removes only that request; cancelling an active crawl restarts one shard and restores its healthy records. Setup deadlines start only when the child owns the crawl; the runtime RPC transport owns the normal 15-second initial budget, while a 60-second recovery deadline quarantines a root that would otherwise pin its shard. A crash-fused shared shard moves affected roots into the bounded quarantine pool, and a second fused quarantine ends those watch lifetimes instead of spawning replacements indefinitely. Every interrupted setup emits overflow after recovery, directory metadata work stays bounded to eight concurrent operations per child, destructive desktop cleanup physically aborts pending native setup, and terminal renderer watches are evicted before error callbacks can retry.",
"oracle": "Fill the four-child healthy pool with multiple roots, inject IPC disconnect-before-exit, queued and active setup cancellation, initial and crash-resubscribe timeouts, and shard-level terminal failure, then assert healthy roots are restored, affected roots spread across no more than four quarantine supervisors, a second quarantine fuse creates no replacement, stale generations stop delivering, and existing same-root clients receive overflow plus later events without error/end. Abort a pending desktop install and require worktree close and app shutdown to settle before the native crawl resolves. Synchronously retry from a renderer terminal-error callback and require a fresh stream that survives the old stream's end. Run eight simultaneous metadata batches in one child and assert no more than eight stats are active there. Drive a real built child through the source supervisor, kill it with SIGSEGV, and require automatic resubscription plus a post-crash event while the harness parent remains alive.",
"invariant": "Local runtime and SSH relay roots use at most four shared watcher children while healthy and at most four additional quarantine children after shard failure. Cancelling queued setup removes only that request; cancelling an active crawl restarts one shard and restores its healthy records. Setup deadlines start only when the child owns the crawl; the runtime RPC transport owns the normal 15-second initial budget, while a 60-second recovery deadline quarantines a root that would otherwise pin its shard. A crash-fused shared shard moves affected roots into the bounded quarantine pool, and a second fused quarantine ends those watch lifetimes instead of spawning replacements indefinitely. Every interrupted setup emits overflow after recovery, event delivery stays bounded, and the SSH relay, PTYs, Git, and other filesystem operations never load the watcher native addon or share its fault boundary.",
"oracle": "Fill the four-child healthy pool with multiple roots, inject IPC disconnect-before-exit, queued and active setup cancellation, initial and crash-resubscribe timeouts, and shard-level terminal failure, then assert healthy roots are restored, affected roots spread across no more than four quarantine supervisors, a second quarantine fuse creates no replacement, stale generations stop delivering, and existing same-root clients receive overflow plus later events without error/end. Abort a pending desktop install and require worktree close and app shutdown to settle before the native crawl resolves. Verify relay build hashes and remote install probes require both relay.js and relay-watcher.js. Launch the built relay with a real PTY and Parcel watch, kill only the watcher child, and require the same relay PID and PTY to answer before a replacement child delivers overflow and a post-crash event.",
"commands": [
"pnpm exec vitest run --config config/vitest.config.ts src/main/ipc/parcel-watcher-process.test.ts src/main/ipc/runtime-watcher-process-pool.test.ts src/main/ipc/parcel-watcher-process-entry.test.ts src/main/ipc/parcel-watcher-event-delivery.test.ts src/main/ipc/filesystem-watcher-local-unsubscribe.test.ts src/main/ipc/filesystem-watcher-wsl.test.ts src/main/runtime/file-watcher-host.test.ts src/main/runtime/rpc/methods/files.test.ts src/main/providers/ssh-filesystem-provider.test.ts src/renderer/src/runtime/runtime-file-client.test.ts",
"pnpm run build:electron-vite",
"node config/scripts/runtime-file-watcher-fault-harness.mjs",
"ELECTRON_RUN_AS_NODE=1 pnpm exec electron config/scripts/runtime-file-watcher-fault-harness.mjs"
"ELECTRON_RUN_AS_NODE=1 pnpm exec electron config/scripts/runtime-file-watcher-fault-harness.mjs",
"pnpm exec vitest run --config config/vitest.config.ts src/relay/relay-filesystem-watch-registry.test.ts src/relay/fs-handler.test.ts src/relay/subprocess.test.ts src/main/ssh/ssh-relay-versioned-install.test.ts config/scripts/package-electron-runtime-contract.test.mjs",
"pnpm run build:relay",
"node config/scripts/relay-watcher-fault-harness.mjs"
],
"testFiles": [
"src/main/ipc/parcel-watcher-process.test.ts",
Expand All @@ -3804,7 +3808,13 @@
"src/main/runtime/rpc/methods/files.test.ts",
"src/main/providers/ssh-filesystem-provider.test.ts",
"src/renderer/src/runtime/runtime-file-client.test.ts",
"config/scripts/runtime-file-watcher-fault-harness.mjs"
"config/scripts/runtime-file-watcher-fault-harness.mjs",
"src/relay/relay-filesystem-watch-registry.test.ts",
"src/relay/fs-handler.test.ts",
"src/relay/subprocess.test.ts",
"src/main/ssh/ssh-relay-versioned-install.test.ts",
"config/scripts/package-electron-runtime-contract.test.mjs",
"config/scripts/relay-watcher-fault-harness.mjs"
],
"assertionRefs": [
{
Expand All @@ -3814,7 +3824,8 @@
"IPC disconnect recovery restores every existing root before accepting new setup",
"queued abort removes one request without restarting healthy roots, while active abort or crawl-scoped timeout restores the shard",
"a crash-resubscribe crawl deadline ends only the stuck root and tears down its shard process",
"missing built entries fail closed and unavailable canary storage remains optional"
"missing built entries fail closed and unavailable canary storage remains optional",
"fault-harness PID reporting cannot clobber an existing filesystem path"
]
},
{
Expand Down Expand Up @@ -3885,24 +3896,82 @@
"a later same-root listener opens a fresh server subscription instead of reusing a dead stream"
]
},
{
"file": "src/relay/relay-filesystem-watch-registry.test.ts",
"assertions": [
"replacement emits overflow before later ordered events",
"terminal shard failure preserves same-root clients through recovery",
"pending setup aborts only after the last same-root client leaves",
"a missing child fails closed even if the relay inherits a VITEST environment"
]
},
{
"file": "src/relay/fs-handler.test.ts",
"assertions": [
"shared watches retain the twenty-root budget and release after every client leaves"
]
},
{
"file": "src/relay/subprocess.test.ts",
"assertions": [
"the real relay bundle serves PTY, filesystem, and Git requests with its adjacent watcher child"
]
},
{
"file": "src/main/ssh/ssh-relay-versioned-install.test.ts",
"assertions": [
"remote completeness requires relay.js, relay-watcher.js, and the install sentinel"
]
},
{
"file": "config/scripts/package-electron-runtime-contract.test.mjs",
"assertions": [
"every release platform packages and gates the hashed watcher child"
]
},
{
"file": "config/scripts/runtime-file-watcher-fault-harness.mjs",
"assertions": [
"the built watcher entry delivers an event before fault injection",
"the harness parent survives watcher-child SIGSEGV",
"the real supervisor automatically replaces the child, resubscribes, and delivers a post-crash event"
]
},
{
"file": "config/scripts/relay-watcher-fault-harness.mjs",
"assertions": [
"the built relay and existing PTY survive watcher-child fault injection",
"replacement emits overflow and a precise post-crash event"
]
}
],
"evidenceRuns": [
{
"date": "2026-07-12",
"runner": "local",
"platform": "macos",
"command": "pnpm exec vitest run --config config/vitest.config.ts src/relay/relay-filesystem-watch-registry.test.ts src/relay/fs-handler.test.ts src/relay/subprocess.test.ts src/main/ssh/ssh-relay-versioned-install.test.ts config/scripts/package-electron-runtime-contract.test.mjs",
"result": "passed",
"durationSeconds": 4.5,
"summary": "Five relay registry, handler, bundled subprocess, install, and package contract files passed with 114 tests."
},
{
"date": "2026-07-12",
"runner": "local",
"platform": "macos",
"command": "node config/scripts/relay-watcher-fault-harness.mjs",
"result": "passed",
"durationSeconds": 0.61,
"summary": "The relay PID and existing PTY survived watcher-child SIGSEGV; overflow and precise events resumed from a replacement child."
},
{
"date": "2026-07-12",
"runner": "local",
"platform": "macos",
"command": "pnpm exec vitest run --config config/vitest.config.ts src/main/ipc/parcel-watcher-process.test.ts src/main/ipc/runtime-watcher-process-pool.test.ts src/main/ipc/parcel-watcher-process-entry.test.ts src/main/ipc/parcel-watcher-event-delivery.test.ts src/main/ipc/filesystem-watcher-local-unsubscribe.test.ts src/main/ipc/filesystem-watcher-wsl.test.ts src/main/runtime/file-watcher-host.test.ts src/main/runtime/rpc/methods/files.test.ts src/main/providers/ssh-filesystem-provider.test.ts src/renderer/src/runtime/runtime-file-client.test.ts",
"result": "passed",
"durationSeconds": 1.73,
"summary": "10 focused supervisor, pool, child-entry, delivery, desktop/WSL cleanup, runtime-host, RPC, SSH ownership, and renderer-client test files passed with 222 tests."
"durationSeconds": 1.7,
"summary": "10 focused supervisor, pool, child-entry, delivery, desktop/WSL cleanup, runtime-host, RPC, SSH ownership, and renderer-client test files passed with 223 tests."
},
{
"date": "2026-07-12",
Expand Down
34 changes: 28 additions & 6 deletions config/scripts/build-relay.mjs
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#!/usr/bin/env node
/**
* Bundle the relay daemon into a single relay.js file per platform.
* Bundle the relay daemon and its crash-isolated watcher child per platform.
*
* The relay runs on remote hosts via `node relay.js`, so it must be a
* self-contained CommonJS bundle with no external dependencies beyond
* The relay runs on remote hosts via `node relay.js`, so both outputs use
* self-contained CommonJS bundles with no external dependencies beyond
* Node.js built-ins. Native addons (node-pty, @parcel/watcher) are
* marked external and expected to be installed on the remote or
* gracefully degraded.
Expand All @@ -17,6 +17,7 @@ const __dirname = import.meta.dirname
// Why: the script lives under config/scripts, so go two levels up to reach the repo root.
const ROOT = join(__dirname, '..', '..')
const RELAY_ENTRY = join(ROOT, 'src', 'relay', 'relay.ts')
const WATCHER_ENTRY = join(ROOT, 'src', 'main', 'ipc', 'parcel-watcher-process-entry.ts')

const PLATFORMS = [
'linux-x64',
Expand All @@ -42,7 +43,22 @@ for (const platform of PLATFORMS) {
outfile: join(outDir, 'relay.js'),
// Native addons cannot be bundled — they must exist on the remote host.
// The relay gracefully degrades when they are absent.
external: ['node-pty', '@parcel/watcher'],
external: ['node-pty', '@parcel/watcher', 'electron'],
sourcemap: false,
minify: true,
define: {
'process.env.NODE_ENV': '"production"'
}
})

await build({
entryPoints: [WATCHER_ENTRY],
bundle: true,
platform: 'node',
target: 'node18',
format: 'cjs',
outfile: join(outDir, 'relay-watcher.js'),
external: ['@parcel/watcher'],
sourcemap: false,
minify: true,
define: {
Expand All @@ -51,9 +67,15 @@ for (const platform of PLATFORMS) {
})

// Why: include a content hash so the deploy check detects code changes
// even when RELAY_VERSION hasn't been bumped (common during development).
// even when RELAY_VERSION hasn't been bumped. Hash both process artifacts
// so a watcher-only change always deploys beside the matching relay host.
const relayContent = readFileSync(join(outDir, 'relay.js'))
const hash = createHash('sha256').update(relayContent).digest('hex').slice(0, 12)
const watcherContent = readFileSync(join(outDir, 'relay-watcher.js'))
const hash = createHash('sha256')
.update(relayContent)
.update(watcherContent)
.digest('hex')
.slice(0, 12)
writeFileSync(join(outDir, '.version'), `${RELAY_VERSION}+${hash}`)

console.log(`Built relay for ${platform} → ${outDir}/relay.js`)
Expand Down
40 changes: 40 additions & 0 deletions config/scripts/package-electron-runtime-contract.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,46 @@ describe('Electron runtime package contract', () => {
)
})

it('packages and release-gates the SSH relay watcher child', () => {
const relayBuild = readFileSync(join(projectDir, 'config/scripts/build-relay.mjs'), 'utf8')
const builderConfig = readFileSync(
join(projectDir, 'config/electron-builder.config.cjs'),
'utf8'
)
const remoteCommands = readFileSync(
join(projectDir, 'src/main/ssh/ssh-remote-commands.ts'),
'utf8'
)
const releaseWorkflow = parse(
readFileSync(join(projectDir, '.github/workflows/release-cut.yml'), 'utf8')
)
const macWorkflow = parse(
readFileSync(join(projectDir, '.github/workflows/release-mac-build.yml'), 'utf8')
)

expect(relayBuild).toContain("'parcel-watcher-process-entry.ts'")
expect(relayBuild).toContain("outfile: join(outDir, 'relay-watcher.js')")
expect(relayBuild).toContain("readFileSync(join(outDir, 'relay-watcher.js'))")
expect(builderConfig).toContain("from: 'out/relay'")
expect(remoteCommands).toContain("joinRemotePath(host, remoteRelayDir, 'relay-watcher.js')")

const assertRelayGate = (steps, publishStepName) => {
const names = steps.map((step) => step.name)
const gate = steps.find((step) => step.name === 'Gate SSH relay watcher process isolation')
expect(gate['continue-on-error']).toBeUndefined()
expect(gate.run).toContain('node config/scripts/relay-watcher-fault-harness.mjs')
expect(names.indexOf('Build app')).toBeLessThan(names.indexOf(gate.name))
expect(names.indexOf(gate.name)).toBeLessThan(names.indexOf(publishStepName))
}

assertRelayGate(releaseWorkflow.jobs.build.steps, 'Publish release artifacts (Linux)')
assertRelayGate(macWorkflow.jobs['build-mac'].steps, 'Publish release artifacts (macOS)')
const releaseNames = releaseWorkflow.jobs.build.steps.map((step) => step.name)
expect(releaseNames.indexOf('Gate SSH relay watcher process isolation')).toBeLessThan(
releaseNames.indexOf('Build Windows release artifacts')
)
})

it('pins the Windows release builder to the VS 2022 runner image', () => {
const releaseWorkflow = parse(
readFileSync(join(projectDir, '.github/workflows/release-cut.yml'), 'utf8')
Expand Down
Loading
Loading