Skip to content

Antigravity runtime temp-dir cleanup starves the server's file I/O for minutes, and successful title generations report a timeout and retry #23877

Open

Description

@luquibu

What happened

On Windows, while I open and use my conversations, the desktop app shows "<host> is reconnecting" and "T3 Code on <host> did not answer". The window stays open and the chats are visible, but the connection to the local backend stays degraded for many minutes. I first saw this on 0.1.1 and reproduced it on 0.1.2-nightly.20261022.3541. Setting UV_THREADPOOL_SIZE=16 (verified in the backend's environment) did not help.

The backend keeps listening on 127.0.0.1:3773. OPTIONS / answers 204 in about 28 ms during the outage, but GET /.well-known/t3/environment times out. In one 5-minute sample: 4× 200, then 20 consecutive 3 s timeouts.

Diagnosis

The outage lasts exactly as long as T3 is deleting an Antigravity runtime's PyInstaller temp directory. My text generation model is Antigravity, so every thread title runs the cycle below. My chat provider is Codex.

  1. ThreadTitleRegenerationService calls AntigravityTextGeneration.generateThreadTitle. AntigravityDriver.makeRuntime creates a per-run run-* directory under <stateDir>/antigravity-tmp/<hash>/. The release step is fileSystem.remove(directory, { recursive: true, force: true }) (apps/server/src/provider/Drivers/AntigravityDriver.ts:223-240).
  2. agy_acp_server.exe unpacks its PyInstaller payload into run-*/_MEI*: at least 6,456 files, counted 17 s after the directory appeared (about 1 GB according to [Bug]: antigravity-acp server orphans 1.16 GB _MEI temp dir per launch, fills OS drive #11204).
  3. The ACP prompt succeeds in 3–7 s. When the runtime scope closes, the recursive remove queues thousands of delete requests on the server's shared libuv pool. During the outage, all 16 pool workers were blocked in NtSetInformationFile(FileDispositionInformationEx) on files under run-*/_MEI*, read at 21:42 UTC during the second cleanup. A runtime snapshot during the first cleanup counted 2,494 pending FSReqCallbacks.
  4. Every other async fs call waits behind those deletes. /.well-known/t3/environment waits on ServerSecretStore.get, a plain readFile, so the descriptor times out, and the client shows reconnecting / did not answer. fs.statSync in the backend returned in 0 ms while an async fs.stat of the same directory did not finish in 2 s.
  5. The runtime scope that holds the temp dir is closed inside Effect.timeoutOption(ANTIGRAVITY_TIMEOUT_MS) (180 s; inner Effect.scoped at apps/server/src/textGeneration/AntigravityTextGeneration.ts:253, timeout at :270). Cleanup time therefore counts against the timeout. A title that was generated successfully is reported as "Antigravity text generation timed out." and thrown away.
  6. Initial titles retry twice (ThreadTitleRegenerationService.ts:126-129), so one title costs three launches and three cleanups. The outbox effect also survives backend restarts: it was on attempt_count 4 of 5 (EffectWorker maxAttempts 5). On restart, removeAntigravityRuntimeTempDirs sweeps the leftover tree at driver creation, which stalls startup the same way. It took 14.5 s on one launch and 49 s on the previous one, and desktop.backendProcess.probeReadiness took 16.8 s and 51.9 s.

On this machine the three cleanups took about 7.7, 9.3 and 4.2 minutes: about 21 minutes of a stalled server for one title. For comparison, fs.rm of 1,000 freshly written small files in a separate Node process took under 1 s. The cost comes from the size of the Antigravity tree. ESET Endpoint Security is installed and may add to it; that was not isolated.

Possible directions, for the maintainers to judge:

  • Do the cleanup outside the timed region, and do not discard a completed generation because cleanup is slow.
  • Delete the tree off the shared libuv pool, or with bounded concurrency, or detached from request-serving work.
  • Avoid one full extraction per text-generation call.

Steps to reproduce

  1. Windows 11, T3 Code desktop 0.1.2-nightly.20261022.3541, Antigravity provider installed and signed in.
  2. Settings → text generation model: Antigravity (gemini-3.8-flash-high here).
  3. Start a new thread with any provider and send the first message. This enqueues thread-title.generate.
  4. Watch %USERPROFILE%\.t3\userdata\antigravity-tmp\<hash>\run-*\_MEI* fill with thousands of files, then get deleted after the prompt returns.
  5. While it is being deleted, poll GET http://127.0.0.1:3773/.well-known/t3/environment: it times out and the app shows "reconnecting". server.trace.ndjson shows session/prompt Success, followed minutes later by AntigravityTextGeneration.generateThreadTitle Failure "timed out", then a retry.

How long each outage lasts depends on how fast the tree is deleted. Not tried on a second machine.

Version

0.1.2-nightly.20261022.3541 (commit 044f072), desktop app with its local backend. Same symptom seen on 0.1.1; the mechanism was not verified there. The t3 triage CLI was also 0.1.2-nightly.20261022.3541.

Environment

Windows 11 Pro 10.0.26200 x64, 22 logical CPUs, 31.5 GiB RAM. Desktop app (Nightly) with local backend (Electron, ELECTRON_RUN_AS_NODE). ESET Endpoint Security running. Chat provider Codex; text generation Antigravity gemini-3.8-flash-high.

Evidence

# Backend libuv workers during the outage (UV_THREADPOOL_SIZE=16), 21:42 UTC, read with GetThreadContext.
# 16 of 16 workers, all the same call; 3 shown:
Wait/Executive  NtSetInformationFile  class=64 FileDispositionInformationEx flags=0x13  ~\.t3\userdata\antigravity-tmp\<hash>\run-QdmTr1\_MEI00007f702\google3\third_party\py\mcp\server\auth\errors.py
Wait/Executive  NtSetInformationFile  class=64 FileDispositionInformationEx flags=0x13  ~\.t3\userdata\antigravity-tmp\<hash>\run-QdmTr1\_MEI00007f702\google3\third_party\py\numpy\f2py\_backends\__init__.py
Wait/Executive  NtSetInformationFile  class=64 FileDispositionInformationEx flags=0x13  ~\.t3\userdata\antigravity-tmp\<hash>\run-QdmTr1\_MEI00007f702\google3\third_party\py\setuptools\_distutils\command\build_py.py
# Control: an idle plain Node process with 16 workers shows them as Wait/Unknown (NtWaitForAlertByThreadId).

# Backend active resources at 21:28:47 UTC (attempt 1 cleanup): FSReqCallback 2494; statSync(userdata) 0 ms; async stat(userdata) >2000 ms

# server.trace.ndjson (UTC)
21:25:01.572 -> 21:48:24.833  ThreadTitleRegenerationService.execute               1403260 ms Success
21:25:01.866 -> 21:33:23.105  AntigravityTextGeneration.generateThreadTitle         501239 ms Failure "Antigravity text generation timed out."
21:25:34.596 -> 21:25:37.862    session/prompt                                        3266 ms Success
21:33:25.109 -> 21:43:29.537  AntigravityTextGeneration.generateThreadTitle         604428 ms Failure "Antigravity text generation timed out."
21:34:04.655 -> 21:34:08.675    session/prompt                                        4020 ms Success
21:43:33.590 -> 21:48:24.804  AntigravityTextGeneration.generateThreadTitle         291214 ms Failure "Antigravity text generation timed out."
21:44:03.226 -> 21:44:09.843    session/prompt                                        6617 ms Success
# ServerSecretStore.get between 21:25 and 21:35: p50 177 ms, max 294323 ms
# environment.metadata.descriptor: repeated 10 s Interrupted from 21:26 to 21:32
# Startup: removeAntigravityRuntimeTempDirs 14549 ms (49124 ms on the previous launch)
# The descriptor returned 200 in 6-9 ms right after the title effect completed at 21:48:24.

Related issues

#11204, #9650 and #12239 reported Antigravity _MEI folders being left behind and filling the disk. This issue is about the cost of deleting the per-run tree while the server is running, not about leftovers. #16932 had the same pool-starvation symptom from pipe reads; that was fixed by #17386, and the code is present in this build. #22586 is a different Windows stall (fff rescan on the event loop).

Fix applied or workaround

Nothing was changed on the machine during triage. Workaround: switch the text generation model away from Antigravity. Raising UV_THREADPOOL_SIZE to 16 did not help.

Filed by

Claude Code (claude-opus-5-5) via t3 triage

Activity

  1. Laguirre-Axoft commented on Oct 23, 2026

    @Laguirre-Axoft

    Same pool starvation, reached through a second path: the managed runtime install/update validation. On this machine it caused an 11-minute "reconnecting / did not answer" outage, and then the update was reported as failed even though validation succeeded.

    What happened

    In Settings → Providers → Antigravity, I started the update from the managed runtime 1.1.1 to 1.3.0. Within seconds, every thread and the Providers page showed "<host> is reconnecting" and "T3 Code on <host> did not answer". The backend kept listening on 127.0.0.1:3773 but GET /.well-known/t3/environment hung. Text generation on this machine is Claude, not Antigravity, so the title path described above was not involved.

    Diagnosis

    1. AntigravityInstallation.install downloads and extracts 1.3.0, then runs validate, which starts the new agy_acp_server.exe in %TEMP%\t3-antigravity-validate-*. PyInstaller unpacks its payload into _MEI* there: 3,426 files in 686 directories, 140 MB.
    2. Validation succeeded in 15.6 s. When its scope closed, the profile directory removal (AntigravityInstallation.ts:356-371) started the recursive delete. It ran at about 10 files/s, and every other async fs call waited behind it, including ServerSecretStore.get and the trace writer.
    3. The validation scope is closed inside the 90 s timeout: validate(...).pipe(Effect.scoped, Effect.timeout(VALIDATION_TIMEOUT)) at AntigravityInstallation.ts:657-666. The cleanup took about 11 minutes, so the install failed with TimeoutError once the delete finished. It never wrote .install-complete.json, the staging directory was removed, and active.json still points at 1.1.1.
    4. The user-facing message is the generic "Could not install Antigravity. Check free disk space and directory access, then try again." Retrying repeats the same outage and the same failure. This matches [Bug]: Windows Antigravity install/reinstall reports failure after successful validation because temp cleanup fails with EBUSY #18551's symptom (successful validation, failed install), with slow cleanup instead of EBUSY as the cause. The retry added by fix(server): Antigravity installs on Windows ARM #18472 does not help here.

    The directions above apply to this path too: run the validation profile cleanup outside the timed region, and keep the delete from occupying the shared pool.

    Steps to reproduce

    1. Windows 11, ESET Endpoint Security running, T3 Code desktop 0.1.2 with its local backend, managed Antigravity runtime 1.1.1 installed.
    2. Settings → Providers → Antigravity, start the update to 1.3.0.
    3. Watch %TEMP%\t3-antigravity-validate-*\_MEI* drain while polling GET http://127.0.0.1:3773/.well-known/t3/environment. It hangs until the directory is gone, and the app shows "reconnecting" the whole time.
    4. When the directory is gone, the endpoint answers again and the install reports the disk space / directory access error. The previous runtime stays active.

    Version

    0.1.2 (tag v0.1.2, commit 044f072), desktop app with its local backend. The t3 triage CLI was 0.1.2-nightly.20261022.3541. AntigravityInstallation.ts is unchanged on main through v0.1.3-nightly.20261023.3553.

    Environment

    Windows 11 Pro 10.0.26200 x64, 22 logical CPUs. ESET Endpoint Security running; Defender real-time protection off. Antigravity managed runtime 1.1.1 (sha 47cb50ee…) updating to 1.3.0.

    Evidence

    # server.trace.ndjson (UTC)
    03:01:23.940                  provider.install.start
    03:01:23.943                  GET https://dl.google.com/agy-extensions/releases/windows/agy-acp-server-1.3.0-windows-x86_64.zip
    03:01:29.978 -> 03:01:45.537  AntigravityInstallation.validate   15.6 s  Success
    03:01:23.940 -> 03:13:10.832  AntigravityInstallation.install   706.9 s  Failure
      AntigravityInstallationError: Could not install Antigravity. Check free disk space and directory access, then try again.
        [cause]: TimeoutError: Operation timed out after '1m 30s'
    
    # ServerSecretStore.get (plain readFile) during the cleanup
    03:01:49.628 -> 03:02:34.489   44.9 s
    03:02:53.1   -> 03:10:40.6    467.5 s  (8 concurrent calls, all the same)
    # environment.metadata.descriptor: 10 s Interrupted, repeatedly, from 03:02:19
    # First 200 again at 03:13:09, in 9 ms, right after the directory was gone.
    
    # Temp dir drain, local sampling
    %TEMP%\t3-antigravity-validate-FMgz2l\_MEI0000a7882: 3,426 files / 686 dirs / 140 MB at start
    3,292 -> 3,076 files in 21 s
    
    # Process sample during the outage
    backend: 16 ms CPU over 5 s; 465 "other" I/O ops per 10 s; 6 threads in Wait/Executive
    ekrn.exe (ESET): 1.42 M read ops, 7.7 s kernel + 2.6 s user CPU per 10 s

    Fix applied or workaround

    Nothing was changed on the machine. The backend recovered by itself when the delete finished. Workaround: do not retry the update from the UI on this machine until the cleanup moves out of the timed region; each attempt costs the same outage and fails the same way.

    Filed by

    Claude Code (claude-opus-5-5) via t3 triage

  2. Laguirre-Axoft commented on Oct 23, 2026

    @Laguirre-Axoft

    Follow-up from the same machine: the slow delete is specific to the T3 Code executable, not to the files, Node, or Electron.

    Same script (fs.promises.rm(dir, { recursive: true, force: true })), same files copied from the live Antigravity _MEI tree, Electron binaries run with ELECTRON_RUN_AS_NODE=1:

    Executable Fresh small files Antigravity _MEI files
    node 22.23.3 665 to 2,614 files/s 10,775 to 11,658 files/s
    node 24.15.0 1,561 files/s 9,432 files/s
    VS Code Code.exe (Electron 43.7.7, Node 24.21.0) 3,170 files/s 10,632 files/s
    T3 Code (Beta).exe 0.1.2 (Electron 44.9.3, Node 24.22.0) 6 to 8 files/s 7 files/s

    The T3 Code rate matches the drain rate seen during the outages, so any recursive delete the backend does on this machine runs about 1,000x slower than the same delete from another process. ESET Endpoint Security 13.1 is the only real-time scanner here (Defender real-time protection off), centrally managed. During the outage ekrn.exe did about 1.4 M read ops per 10 s. Both binaries are validly signed. Not confirmed from ESET's own logs, so the exact module is unknown. Users on managed endpoints with similar policies will see the same cost, which makes doing the cleanup outside the shared pool and outside timed regions more important.

    Filed by Claude Code (claude-opus-5-5) via t3 triage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions