Skip to content

[Bug]: On Windows, every git start blocks the server's event loop, freezing it for seconds #25245

Open

Description

@3mdistal

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. On Windows with Git for Windows in C:\Program Files\Git, time the synchronous part of starting
    git.exe from Node, one start a second, the way the server's sweeps spread them out:

    // node time-git-spawn.cjs
    const { spawn } = require("node:child_process");
    const git = String.raw`C:\Program Files\Git\mingw64\bin\git.exe`;
    let runs = 0;
    const start = () => {
      const before = performance.now();
      spawn(git, ["--version"], { stdio: "ignore", windowsHide: true });
      console.log(`${(performance.now() - before).toFixed(0)} ms`);
      if (++runs < 12) setTimeout(start, 1000);
    };
    start();

    At times, back-to-back starts were fast while spaced ones stayed slow (see below), so a
    benchmark that starts git back to back can miss this.

  2. Run the T3 Code server on the same machine with a few projects whose threads sit on branches,
    and leave it running. Pull request discovery and status checks start git processes every minute.

  3. Watch event-loop-stalls.ndjson in the server's logs, or use the app while a sweep runs.

Expected behavior

Starting a git process doesn't hold the event loop long enough to notice. Websocket requests,
provider events and the UI stay responsive while background sweeps run.

Actual behavior

On the affected PC, how long a synchronous spawn() of the installed git.exe takes varies over
time:

  • One afternoon, in interleaved back-to-back runs of 30-60 starts per program, each waiting for
    the previous one to exit, it typically took 486-582 ms per start (worst 2.1-2.3 s). A
    byte-identical copy of git.exe and its four DLLs in a fresh folder took 10-14 ms (worst
    217-390 ms).
  • That evening, the same back-to-back runs took 4-5 ms for the original and the copy alike, while
    starts spaced a second apart still took 0.5-2 s. That is why the snippet spaces its starts.

Hard links and the program's location were checked and ruled out. So were Smart App Control (off),
attack surface reduction rules (none) and Image File Execution Options entries (none for
git.exe or node.exe). Antivirus scanning of the heavily used original would fit the data, but
that is not proven. A short trial excluding the file from Defender didn't settle it: the
back-to-back starts had already got fast before the exclusion went on, and the server kept
freezing while the file was excluded, with stalls of 5.3 s and 2.1 s.

The server starts git often: in a 27-minute trace, pull request sweeps alone started 11,182
processes, 11,180 of them git, about 400 a minute. Every start runs on the event loop, so
overlapping starts queue there. 60 git reads at the git semaphore's concurrency of 8 stalled the
event loop for up to 4.4 s and 4.5 s in two rounds. While that happens the app freezes, and
websocket requests and provider events wait behind the stall.

Impact

Major degradation or frequent failure

Version or commit

Freezes seen on T3 Code (Nightly) 0.1.3-nightly.20261024.3643. Git still starts through Node's
spawner on the event loop on main @ 7bafc40.

Environment

Windows 11 Pro (build 26300), Git for Windows 2.55.0 in C:\Program Files\Git, Microsoft
Defender on. The timings above used Node 24.22.1.

Workaround

None that holds up. A fresh copy of git.exe was faster only in the afternoon runs, and pointing
T3 Code at a copy is fragile. Settling or archiving threads whose pull requests are merged
lowers how many git processes the sweeps start (#24098), not what each start costs.

Activity

  1. Enmani commented on Oct 26, 2026

    @Enmani

    Independent Windows evidence: native spawn stalls and a local-primary heartbeat timeout

    I can independently confirm the native-spawn blocking symptom, with a sampled backend stack and a separately captured local connection failure. Posting here rather than opening a duplicate.

    Environment / build disclosure: Windows 11 Pro, build 26300; desktop executable reports T3 0.1.2; shipped Electron 44.9.3 / Node 24.22.0; Git 2.40.1.windows.1. The server bundle is locally modified with asynchronous Windows executable lookup and SQLite read routing. This is not a clean-stock reproduction. Its packaged Effect spawner still calls native NodeChildProcess.spawn(...) directly on the backend event loop. The user regularly runs dozens of agent threads, but the exact active count during this capture was not established.

    Observed evidence

    A 100 Hz V8 profile of the live backend on 2026-10-26, approximately 01:20:59–01:22:00 UTC:

    Measurement Result
    Profile duration 61,081.8 ms
    Native spawn sampled wall time, summed across its leaf nodes 46,298.9 ms (~75.8%)
    Longest consecutive run of samples in native spawn 7,381.2 ms
    runGitCommand spans overlapping the capture 41
    processRunner.runProcessCore spans marked process.command=git, overlapping the capture 7

    The sampled call path was:

    native spawn
      node:internal/child_process -> spawn
      node:child_process -> spawn
      @effect/platform-node-shared/NodeChildProcessSpawner.js
        -> NodeChildProcess.spawn(command.command, command.args, spawnOptions)
      Effect runtime
    

    The native call site was verified in the running installation's server bundle. The public process runner still delegates launch to the Effect spawner.

    These are statistical wall-time samples, not CPU utilization: they can include native waiting/descheduling. Consecutive samples in the same frame may cover back-to-back calls, so 7.38 seconds is not claimed to be one measured spawn call. The Git spans overlap and are not an exact or additive process-launch count. Profiler startup acknowledgement time is not included in the spawn attribution.

    To check whether slow launch exists without profiling, a separate finite fixture using the same shipped Electron executable launched only git --version:

    Phase Synchronous spawn return, launch 1 / 2 Total command elapsed, launch 1 / 2
    No profiler 621.9 / 1,028.0 ms 3,180.9 / 3,858.6 ms
    100 Hz profiler 656.8 / 411.2 ms 2,338.8 / 3,536.9 ms

    This is only two launches per phase under uncontrolled host load, not a benchmark proving zero profiler effect. It does show that slow synchronous launch also exists with the profiler off.

    Real disconnect: failure started before focusing T3

    In a later window, with no CPU capture active, an out-of-process probe and foreground-state sampling captured:

    UTC Observation
    01:39:46 / 01:39:49 Local metadata endpoint responded in ~16 ms
    01:39:59.434 TCP connected immediately, but HTTP response headers timed out after ~828 ms
    01:40:03 / 07 / 10 Repeated header timeouts, immediate TCP connection
    01:40:12.887 User switched from another window to the already-open T3 window
    01:40:18.766 EnvironmentSupervisor.connectionLost, local primary target, transport failure; detail classified as stopped responding for ...
    After switching away again Independent probe failures continued

    The client ping timeout is 15 seconds, and the disconnect path distinguishes this ping-timeout detail.

    For this event, backend response failure preceded foreground return by 13.45 seconds. The observed “Syncing messages…” then “reconnecting” sequence therefore does not establish that focusing the window caused the failure.

    The metadata endpoint also performs asynchronous filesystem work, so its timeout alone is not a pure event-loop measurement. The CPU profile and disconnect are different windows; we have not proven that a particular Git launch caused this exact heartbeat loss.

    Suggested T3-side investigation / changes

    1. Isolate native child creation from the event loop serving RPC and heartbeats. Evaluate a long-lived launch worker/broker rather than merely wrapping spawn() in a Promise/Effect. Preserve stdout/stderr streaming, backpressure, exit status, cancellation, and process-tree ownership. A launch worker may block on Windows, but the primary backend should remain responsive.
    2. Reduce duplicate background Git launches per worktree / branch. Coalesce in-flight read requests, reuse valid cached results, and stagger sweeps. Keep freshness checks before settlement or other state-changing decisions. This complements [Bug]: Threads whose branch PR is merged or closed re-read git state every minute #24098 and should not require lowering user agent-task concurrency.
    3. Measure creation delay separately from command runtime and queue wait. Record bounded aggregates for native spawn return latency, pending launches and event-loop lag. Increasing RPC concurrency or extending the 15-second timeout alone does not remove synchronous blocking.
    4. Test responsiveness under deliberately delayed child creation. Assert that independent RPC / heartbeats remain responsive while the launch worker is delayed, and that command output, cancellation and cleanup retain their behavior.

    A small spawn-host direction appeared in #18219, but that PR is closed/unmerged and described Linux support; I have not verified a Windows equivalent. #21289 concerns Linux fork/RSS cost, which should not be assumed to explain Windows native creation latency.

    The underlying Windows cause remains unknown; these measurements do not establish Defender, disk, memory growth, or a T3 database lock as the cause. No raw profiles, conversation contents, credentials, environment-variable values, private repository paths or host identifiers are attached.

    Prepared and submitted by Codex at the user's request. No running T3 tasks were stopped or restarted for this investigation.

  2. added 8 commits that reference this issue on Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions