Antigravity health probe filled my C: drive (56 GB of _MEI orphans); pre-#12008 leftovers are never cleaned #12239

Activity

1. juliusmarminge commented on Sep 17, 2026

   

   juliusmarminge

   on Sep 17, 2026

   Member

   More actions

   Triage

   Confirmed on current main (dd9528a97). The ongoing leak is already fixed on nightlies that include #12008. The leftover reclaim this issue asks for is not implemented, and that omission was deliberate. Not a duplicate of closed #9650.

   What happens

   On a nightly from before #12008, the Antigravity health probe launched the real ~430 MB agy_acp_server.exe PyInstaller bundle about once a minute (providerHealthRefreshInterval default is 1 minute) plus once at boot. Each launch unpacked ~1 GB into %LOCALAPPDATA%\Temp\_MEI*. The probe was force-killed, so the bootloader never cleaned up. That matches this report (51 dirs / ~56 GB in a day, no Antigravity sessions in the event log, agy --version leaves no orphan) and the pile of #9650 data points (30 GB–741 GB).

   8c18b5bb2 already stopped that:

   • The probe no longer spawns. It only installation.resolves the on-disk binary and returns a synthetic agentInfo.version. Sessions and manual refreshes still spawn.

         // The probe must not spawn. The agent is a PyInstaller one-file bundle
         // that unpacks about 1 GB per launch, and the health check runs every
         // minute. Resolving the install on disk is enough to report installed
         // and version. The response below is synthetic: only agentInfo.version
         // is read from it. Sessions and manual refreshes still spawn.

   • Each real ACP process gets its own temp dir under the profile (TEMP/TMP on Windows, TMPDIR elsewhere), removed when the runtime scope closes.
   • On driver create, removeAntigravityRuntimeTempDirs deletes the profile temp root only.

    * Removes every per-process runtime temp directory under the profile. Call
    * once when the driver starts, before it launches any process, so a previous
    * server that was killed mid-session cannot leave unpacked runtimes behind.
    * Only the profile-owned directory is touched. The system temp directory
    * belongs to other programs and Windows does not lock data files, so sweeping
    * it could gut a live extraction.

   Nothing in main after that commit walks %TEMP%\_MEI*. Pre-#12008 orphans stay on disk until the user deletes them. Updating does not reclaim the 56 GB.

   Ship status

   Build	Probe spawn	Profile isolation	System-temp leftover sweep
   Stable v0.0.42	still leaks	no	no
   Nightlies from 0.0.41-nightly.20260916.1795 / 0.0.43-nightly.20260916.1811 (contains 8c18b5bb2)	stopped	yes	no
   Current main (dd9528a97)	stopped	yes	no

   v0.0.42 does not contain #12008. Anyone still on that stable (or an older nightly) is still leaking. This reporter was on a pre-#12008 nightly; after they update, the flood stops and the old dirs remain.

   Why #12008 dropped the sweep

   #11657 (closed, commits imported into #12008) had cleanOrphanedAntigravityTempDirs: on Windows it listed %TEMP%, deleted every _MEI* that contained google3 or google, and ignored EBUSY/EPERM. Theo removed that path on purpose: Windows locks loaded executables, not data files, so deleting another program’s live _MEI can gut it. The reporter’s constraints are exactly the gap that decision left: agy markers, age, no live process, never blanket-delete _MEI*. #11657’s google3/google check is still too wide for a start-of-driver sweep.

   Not a duplicate

   • #9650 (closed by fix(antigravity): stop health checks from filling the disk with _MEI folders #12008) is the leak itself. This issue is the residual: T3 created the dirs and never takes them back.
   • #12206 is a Windows session-start -1 after recent changes (needs more info). Disk-full leftovers can cause that, but it is a different symptom. Do not merge them.

   No other open issue asks for a safe system-temp reclaim.

   Likely code

   • apps/server/src/provider/acp/AntigravitySessionFiles.ts — removeAntigravityRuntimeTempDirs only deletes the profile temp root.
   • apps/server/src/provider/Drivers/AntigravityDriver.ts — that sweep on create.
   • apps/server/src/provider/Drivers/AntigravityDriver.ts — no-spawn probe (already shipped).
   • apps/server/src/provider/antigravityAuthSupport.ts — TEMP/TMP redirected under the profile for new processes only.

   Fix direction

   Do not restore #11657’s _MEI* + google3 sweep as a boot-time wipe. Keep #12008’s rule: never touch unmarked system temp.

   A Windows-only, best-effort reclaim of T3-created leftovers is the remaining work:

   1. Consider a %TEMP%\_MEI* dir only if it has Antigravity markers (agy_acp_licenses.txt and/or localharness / google3/third_party/jetski_prod/localharness). Skip everything else.
   2. Skip dirs newer than a few days, and skip any dir that still has an open handle / live process (ignore EBUSY/EPERM; do not retry-loop at probe cadence).
   3. Run once per driver create (or a single first-run after upgrade), not every health tick.
   4. Tests: marked stale dir is removed; unmarked _MEI*, young marked dir, and a dir that fails with EBUSY are left alone. Do not hit a real system temp in CI — inject the directory.

   Manual cleanup in the Antigravity user doc is optional and is not a substitute.

   Workaround

   1. If you are on v0.0.42 or a nightly from before 2026-09-16: update to a nightly that includes fix(antigravity): stop health checks from filling the disk with _MEI folders #12008 (0.0.43-nightly.20260916.1811 or later), or set Settings → Providers → Advanced → Provider health check interval to 0 until you can. Otherwise new orphans keep appearing.
   2. Fully quit T3 (and any other PyInstaller app you care about).
   3. Delete only _MEI* dirs under %LOCALAPPDATA%\Temp that contain agy_acp_licenses.txt or localharness. Do not wipe every _MEI*.
   4. After fix(antigravity): stop health checks from filling the disk with _MEI folders #12008, new session extracts live under the provider profile and are swept on the next driver start. They should not refill %TEMP%.

   Accepting as a server / Antigravity leftover-cleanup bug. The probe leak is done; reclaiming the pre-fix %TEMP%\_MEI* pile is not.

   👍React with 👍3Reacted by kamkm, Sunjay Kumar and CHRIS
2. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   acceptedfeature request acceptedfeature request accepted

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Sep 17, 2026
3. criticalstrike18 commented on Sep 20, 2026

   

   criticalstrike18

   on Sep 20, 2026

   More actions

   Confirmation & Technical Details on Windows 11 (v0.0.42 Stable)

   Confirming this issue also occurs on Windows 11 with T3 Code v0.0.42 (stable release), where it completely exhausted a C: drive (dropping free space from >100 GB to < 250 MB) with ~90-100 orphaned Temp\_MEI* folders totaling over 100 GB.

   Root Cause Analysis

   In v0.0.42 (prior to #12008 reaching a stable release), AntigravityProvider creates disposable ACP runtimes (makeDisposableRuntime / refreshModels / provider health check). When the effect scope closes or times out, the child process is terminated via Windows TerminateProcess().

   Because Win32 TerminateProcess() terminates the process abruptly at the OS level:

   1. PyInstaller's runtime bootloader never executes its exit cleanup hook (atexit).
   2. The extracted directory in %LOCALAPPDATA%\Temp\_MEI* (~1.16 GB per extraction) is permanently orphaned on disk.

   Recommendations

   1. Hotfix / Backport: Strongly recommend backporting fix(antigravity): stop health checks from filling the disk with _MEI folders #12008 to a v0.0.42.x stable patch. Stable users who do not run nightly builds continue to face rapid disk exhaustion from routine provider health checks.
   2. Legacy Cleanup: As suggested in the issue description, include a safe one-time or startup sweep of unmanaged legacy _MEI* folders in %TEMP% that match Antigravity signatures (localharness_external.exe / agy_acp_licenses.txt) and are not locked by any active process.

   👍React with 👍1Reacted by kamkm
4. cakeseyboy commented on Sep 24, 2026

   

   cakeseyboy

   on Sep 24, 2026

   More actions

   Confirmed on Windows 11 with T3 Code (Alpha) v0.0.42 stable.

   • Antigravity was enabled with the performance profile (providerHealthRefreshInterval: 60000). server.trace.ndjson recorded checkAntigravityProvider starts at 22:23:33, 22:25:37, 22:27:32, 22:29:25, and 22:31:21 local time on 2026-09-23. New %LOCALAPPDATA%\Temp\_MEI* directories appeared at the same times (within one second).
   • A live process had the parent chain T3 Code (Alpha).exe -> T3-managed agy_acp_server.exe. A representative extracted directory contained agy_acp_licenses.txt and google3\third_party\jetski_prod\localharness.
   • I measured 26 remaining _MEI* directories at 28.09 GiB; the large ones were about 1,192 MiB each. Earlier there were 92 _MEI* directories and C: had essentially no free space. The folder count subsequently fell while free space recovered past 100 GiB.
   • After disabling the Antigravity provider, no new extraction directories appeared during the next health-check cycles.

   This independently confirms the pre-#12008 health-probe leak is still present in stable v0.0.42. A stable backport and safe cleanup of marked legacy orphans would address the two remaining user-facing problems.
5. sami999khan999 commented on Sep 24, 2026

   

   sami999khan999

   on Sep 24, 2026

   More actions

   Still reproducing on T3 Code (Alpha) 0.0.42 (Windows 11 Pro 26200) — new orphans are being created right now, not just legacy leftovers, so #12008 doesn't seem to cover this path (and #12717's 2-day sweep wouldn't keep up).

   What I saw (2026-09-24):

   • Antigravity ACP was downloaded to ~\.t3\tools\antigravity-acp\win32-x64\versions\47cb50ee…\agy_acp_server.exe (411 MB) at 09:09.
   • From 09:26 to 10:38, 29 _MEI* folders (~1.16 GB each, ~34 GB total) accumulated in %LOCALAPPDATA%\Temp (the root of Temp, not under a T3 profile dir) and filled C: to 0 bytes free.
   • Contents are the Antigravity bundle (google3, grpc, googleapiclient, pydantic_core, websockets, …).

   Correlation (process-start log, 400 ms polling, so detection lags slightly):

   10:51:58.10  _MEI0000228c2 created
   10:51:58.90  agy_acp_server.exe detected, parent = T3 server (T3 Code (Alpha).exe …\server.asar\apps\server\dist\bin.mjs)
   10:53:31.28  taskkill.exe x2, parent = T3 server
   10:53:35.45  T3 Code (Alpha).exe child spawned by T3 server -> _MEI000037bc2 created the same instant
   

   Earlier I also caught the server running taskkill /pid <n> /T /F. Force-killing the PyInstaller onefile bootloader's tree means it never gets to remove its _MEI dir, so every restart/kill of agy_acp_server.exe leaks ~1.16 GB. After I deleted everything, new folders appeared within minutes.

   Suggestions:

   • Stop the ACP server gracefully (close stdin / CTRL_BREAK / wait) before falling back to taskkill /F, or avoid restarting it as often.
   • Give the child a dedicated TMP/TEMP dir and delete it after the process exits — reliable even after a force-kill.
   • Sweep Antigravity-marked _MEI* dirs that aren't locked on startup/periodically, not only ones older than 2 days.

   Workaround for others: delete %LOCALAPPDATA%\Temp\_MEI* (in-use ones are locked and will be skipped).
6. dotcomdomain commented on Sep 26, 2026

   

   dotcomdomain

   on Sep 26, 2026

   More actions

   Confirming on T3 Code (Alpha) 0.0.42 stable, Windows — same signature, still actively leaking (not just legacy leftovers).

   What I measured (2026-09-25/26):

   • Deleted ~90 GB of %LOCALAPPDATA%\Temp\_MEI*\ orphans; ~72 GB re-accumulated within days.
   • Each orphan: ~1.164 GB, 8,247 files. Contents: \google3/third_party/jetski_prod/localharness, \websockets, \grpc, \googleapiclient, \pydantic, \matplotlib\ data — the Antigravity bundle.
   • \unleash-repo-schema-v1-codeium-language-server.json\ also present in Temp root (fetched 2026-09-25).
   • No live process holds them (all orphans); \�gy.exe\ (Go CLI, no version metadata) runs fine with no _MEI of its own — consistent with the health-probe-spawns-\�gy_acp_server.exe\ path, not the CLI itself.

   Impact beyond disk-full: this is 100+ GB of write/delete churn on the C: SSD every few days, plus disk-full events (drive dropped to near-zero free). Supporting the asks already here: stable backport of #12008 (nightly-only fix leaves stable users like me still leaking), plus the guarded reclaim of marked stale orphans.
7. piotrnowakowski commented on Sep 27, 2026

   

   piotrnowakowski

   on Sep 27, 2026

   More actions

   Adding another data point, because this is a severe issue that needs a fix in stable, not only in nightlies. It quietly fills the system drive of any Windows user who has Antigravity enabled, even if they never send it a single prompt.

   Environment: T3 Code (Alpha) 0.0.42 stable, Windows 11 Pro 10.0.26200, Antigravity provider enabled (oauth-personal), default health refresh.

   Impact:

   • C: drive (1.9 TB) down to 5.6 GB free.
   • 324 orphaned %LOCALAPPDATA%\Temp\_MEI* dirs, ~370 GB, all created between 2026-09-23 and 2026-09-27. The count grew from 319 to 324 during a single ~15-minute investigation.
   • Each dir is ~1.2 GB; ~880 MB of that is google3\third_party\jetski_prod\localharness\localharness, plus googleapiclient (~87 MB), python310.dll, and agy_acp_licenses.txt.

   Why it happens (from server.trace.ndjson):

   • checkAntigravityProvider (parent: applySnapshot) runs about every 5 min 10 s while T3 is open, e.g. 2026-09-27 at 15:00:38, 15:05:47, 15:10:56, 15:16:05, 15:41:14.
   • Each check goes AntigravityDriver.makeRuntime → makeAntigravityAcpRuntime, which launches ~\.t3\tools\antigravity-acp\win32-x64\versions\47cb50ee…\agy_acp_server.exe (430 MB PyInstaller one-file). The span takes ~8.8 s and then the process is force-terminated.
   • PyInstaller's bootloader never gets to remove its _MEI extraction dir, so every health check leaks ~1.2 GB.
   • The _MEI creation-hour histogram matches the probe cadence exactly: 5 probes in the 15:00 hour and 5 new dirs in that hour, ~8–13 dirs/hour on busy days. That is ~14 GB per hour of T3 simply being open.

   Ask:

   1. Don't spawn the full ACP runtime for a status probe. Use a lightweight check (binary present + auth record) or reuse a long-lived process.
   2. If a process must be spawned, shut it down gracefully (close stdin / ACP shutdown) so PyInstaller can clean up, or set runtime_tmpdir / a fixed extraction dir.
   3. Ship the leak fix and a one-time leftover sweep in stable. Users on 0.0.42 are losing hundreds of GB without any visible signal. As it stands, the only mitigation is to disable Antigravity and delete _MEI* manually.