perf(host): melo-desk-001 process creation costs 1.4 to 4 s system-wide; 3.81M leaked kernel Token objects (~10 GB paged pool) since boot

[kyle-sexton]

Note

Host-specific: melo-desk-001 (Windows 11 Pro 10.0.26200, Intel Core Ultra 9 285K, 24 logical CPUs, 64 GB). Not a plugin defect. Filed so the diagnosis captured on 2026-09-04 is not lost, and so the follow-up (reboot, then attribute the leaker from an admin shell) can run in a fresh session. Needs a human with an elevated shell; not agent-ready.

Symptom

Every process creation on the host costs 1.4 to 4 s, system-wide, at 6 to 7% CPU load with 26 to 30 GB of RAM free. This is not Claude Code churn: it reproduces from a bare PowerShell with nothing else on the critical path.

probe (2026-09-04, ~03:30 to 12:30 local)	measured	this host's documented healthy floor (claude-ops/skills/audit-performance/reference/known-performance-issues.md)
cmd /c exit 0 via [Diagnostics.Process]::Start, absolute path, no PATH search	1,339 to 1,573 ms	206 ms
jq -n 1 from PowerShell (absolute path)	1,599 to 2,546 ms	not recorded
bash -c true (Git Bash usr/bin/bash.exe)	3,555 to 4,347 ms	123 ms drained, 1,144 ms under a 13-session storm
$(echo hi) subshell (MSYS fork, no exec)	769 to 2,271 ms	not recorded
claude --version	4,113 ms	not recorded
claude plugin list	5,780 ms	not recorded

Split of one cmd.exe creation with CREATE_SUSPENDED (P/Invoke CreateProcess, then ResumeThread + wait): kernel-side create 668 ms, resume-to-exit 683 ms. For jq.exe: 1,488 ms + 1,058 ms. The kernel half alone is 3 to 7x the whole healthy spawn, so the cost is inside process creation itself, not in DLL loading or the console layer.

Consequence measured on the plugin fleet: one claude-ops fleet-state.sh run (736 process creations at the time) took 17.0 minutes; the pre-rewrite --all mode took 51 minutes; each Bash tool call in a Claude Code session carries 30 to 60 s of overhead. PR #3714 cuts the script to 12 creations, but the per-creation floor is the host's.

Kernel state (the lead)

Kernel object-type table via NtQueryObject(NULL, ObjectTypesInformation):

type	objects	handles	high-water objects
Token	3,811,482 → 3,813,997 within an hour	1,739	equal to current (never freed since boot)
Key	189,005	189,011	189,595
File	170,466	18,926	279,542
Event	67,560	72,546
EtwRegistration	50,926	50,926
Section	30,744	10,046	70,603
Process	837 to 901 (\Objects\Processes) vs 611 to 664 visible		1,364

Pool: paged 9,894 MB, nonpaged 2,452 MB (\Memory\Pool Paged Bytes, \Memory\Pool Nonpaged Bytes). 3.8M tokens at a realistic 2 to 3 KB each accounts for the paged pool. Token growth rate at "idle" (other sessions running): 6.6 to 13.9 tokens/s; 20 cmd /c exit spawns did not raise it beyond background, so the minter is not per-process-creation but something continuous (impersonation-shaped, e.g. RPC/ALPC or a driver IOCTL path).

Uptime at capture: 2 days 0 h 55 min (boot 2026-09-02 02:53:19Z). No Resource-Exhaustion-Detector or pool events in the System log.

What was ruled out (measured, not assumed)

• Session churn / spinning shells. 190 to 214 bash.exe and 65 to 71 conhost.exe alive; 43 bash are direct children of 6 claude.exe sessions, ~53 are orphans. Four bash.exe running skill-quality/scripts/check-skill.sh --require-evals <legacy|newbie> from /tmp/tmp.* test copies have been spinning since Sept 2 to 3 with no child processes (PIDs 25092 at 8,391 CPU-s, 12704 at 5,409, 61144 at 2,333, 55592). Suspending all four with NtSuspendProcess moved the spawn floor from 2,978 to 2,514 ms (15%). Resumed afterwards. They are victims and a separate defect (a shell loop in check-skill.sh that never terminates on this host), not the cause.
• ASUS ADU.exe (PID 2516, user session, elevated). Holds 174,378 handles, 173,808 of one type (registry Key). Suspending it changed nothing. A real leak, but not this one.
• Antivirus / EDR. WinDefend, Sense, WdNisSvc all Stopped; WdFilter not loaded; no third-party security process or minifilter visible (fltmc needs admin). Get-MpComputerStatus: provider load failure.
• WDAC / Smart App Control / AppLocker / AppInit / IFEO. Kernel CI enforced (HVCI), user-mode CI off, Smart App Control 0, no AppLocker rules, LoadAppInit_DLLs 0, IFEO not readable without admin.
• PATH. 36 entries, all present, none UNC, longest probe 14 ms. Absolute-path CreateProcess is equally slow.
• Desktop heap. SharedSection=1024,20480,768 (defaults).
• AppCompat. 12 custom shims, 208 PCA store entries. Not unusual.
• Tracing tools. No procmon/sysmon/wpr/xperf running; logman query -ets shows no sessions (unprivileged view).

Non-Microsoft kernel drivers running: ASUS AsIO3.sys, IOMap64.sys; MICSYS MsIo64.sys; Creative CtiAIo64.sys; NVIDIA nvlddmkm, nvhda64v, nvvad64v; Intel platform/graphics/NPU set; Realtek audio/NIC/BT. The three raw-I/O drivers (AsIO3, IOMap64, MsIo64) are polled continuously by Armoury Crate / lighting services, which is the profile that fits a per-IOCTL token or reference leak, but this is a hypothesis, not a measurement.

Reproduce the measurements (no admin needed)

Spawn floor:

$sw=[Diagnostics.Stopwatch]::StartNew(); 1..5 | % { $psi=[Diagnostics.ProcessStartInfo]::new('C:\Windows\System32\cmd.exe','/c exit 0'); $psi.UseShellExecute=$false; $psi.CreateNoWindow=$true; $p=[Diagnostics.Process]::Start($psi); $p.WaitForExit() }; "$([int]($sw.ElapsedMilliseconds/5)) ms/spawn"

Token object count (compile once per shell):

Add-Type -TypeDefinition @'
using System; using System.Runtime.InteropServices;
public static class ObjTypes { [DllImport("ntdll.dll")] static extern int NtQueryObject(IntPtr h, int cls, IntPtr buf, int len, out int ret);
  public static long Count(string type) { int len=1024*1024; IntPtr buf=Marshal.AllocHGlobal(len); int ret; if (NtQueryObject(IntPtr.Zero,3,buf,len,out ret)!=0) return -1;
    int n=Marshal.ReadInt32(buf); IntPtr p=buf+8; long r=-1;
    for (int i=0;i<n;i++){ ushort nl=(ushort)Marshal.ReadInt16(p,0); ushort nm=(ushort)Marshal.ReadInt16(p,2); string name=Marshal.PtrToStringUni(Marshal.ReadIntPtr(p,8),nl/2); if(name==type){ r=(uint)Marshal.ReadInt32(p,16); break;} p=p+(0x68+((nm+7)&~7)); }
    Marshal.FreeHGlobal(buf); return r; } }
'@
[ObjTypes]::Count('Token'); Start-Sleep 20; [ObjTypes]::Count('Token')

Pool: Get-Counter '\Memory\Pool Paged Bytes','\Memory\Pool Nonpaged Bytes','\Objects\Processes'.

Next steps (in order)

1. Capture the pool tag before rebooting, from an admin shell: poolmon -b (WDK) or Process Explorer's kernel memory view; expect Toke to dominate paged pool. If WinDbg is available: !poolused 2 Toke and !token on a sample. This is the one measurement that cannot be taken after a reboot.
2. Reboot. Then re-run the spawn-floor probe. Expected: cmd /c exit back near 200 ms and bash -c true near 120 ms. If not, the leak is not the cause and this issue needs a different lead.
3. Attribute the minter after the reboot, from an admin shell: sample [ObjTypes]::Count('Token') over 60 s, then stop one candidate at a time and re-sample: ArmouryCrateService, LightingService (Aura), ROG Live Service, AsusFanControlService, AsusUpdateCheck, asComSvc, then Razer (RzBTLEManager), NVIDIA (nvcontainer), Wispr Flow. A rate that drops to ~0 names the culprit. Growth that persists with all of them stopped points at a Windows component (CloudAP / AzureAD-joined token path is the next suspect; the account is AzureAD\KyleSexton).
4. Kill the four spinning check-skill.sh shells (PIDs above; verify command lines first, they belong to old sessions) and file the loop as its own skill-quality defect with the --require-evals legacy / newbie fixture names.
5. Optional: ADU.exe's 174K registry-key handle leak is worth an Armoury Crate update or removal on its own.

Related

• perf(hooks): Windows process-creation tax makes the hook set unusable - 1,778 timeouts / 20.6 hours blocked over 9 days #3508 (Windows process-creation tax on the hook set), Hook performance program: measure and cut every runtime-cost surface on the Windows host #3623 (hook-performance program), PR perf(claude-ops): fleet-state.sh in three jq passes, 12 process creations instead of 736 #3714 (fleet-state.sh at 12 process creations, measured on this host in this state).
• Another session's read of the same slowness attributed it to six Claude Code sessions spawning 144 shells in 30 minutes. That churn is real but secondary: the per-creation floor is 10 to 25x this host's own healthy baseline with no storm present, and suspending the busiest shells moved it 15%.

[kyle-sexton]

[kyle-sexton]

Post-reboot readings (2026-09-05, 01:15 to 01:40Z)

The host rebooted at 2026-09-04T23:27:02Z. Step 2's prediction holds: the spawn floor is back, and the leak is still minting.

probe	before reboot (2 d uptime, from the report above)	after reboot
cmd /c exit 0 via [Diagnostics.Process]::Start, absolute path	1,339 to 1,573 ms	14 ms
paged pool	9,894 MB	1,049 MB at 1 h 51 min; 1,231 MB at 2 h 10 min
nonpaged pool	2,452 MB	1,493 to 1,532 MB
Token objects	3,811,482 (high-water equal)	23,405 at 1 h 51 min; 37,144 at 2 h 01 min; 70,380 at 2 h 10 min
Token growth, 60 s window	6.6 to 13.9/s	14.98/s
Token growth, 3 s window		0/s (minting is bursty; short windows under-read)
Token growth, since-boot average	about 22/s	3.7/s at 1 h 57 min, 8.9/s at 2 h 10 min and rising
ADU.exe handles	174,378	454 (pid 9052)
bash.exe / conhost.exe alive	190 to 214 / 65 to 71	10 / 16, with 5 claude.exe sessions

At the since-boot average the count re-crosses 250,000 in well under a day, and the two-day level returns in days, not weeks. The reboot bought time; only attribution ends it.

Step status

1. Pool tag before reboot: missed. The reboot happened before anyone ran poolmon. Next time the host is in the degraded state, capture it first; it is the one measurement a reboot destroys.
2. Reboot: done, and it confirmed the lead (14 ms floor, about 1 GB paged pool).
3. Attribute the minter: operator follow-up, needs an elevated shell. Every candidate service is Running again: ArmouryCrateService, LightingService, ROG Live Service, AsusFanControlService, AsusUpdateCheck, asComSvc, AsusCertService, the three Razer Chroma SDK services, Razer Game Manager Service 3, NvContainerLocalSystem, NVDisplay.ContainerLocalSystem. The bisection script below is paste-ready.
4. Spinning check-skill.sh shells: gone with the reboot. The loop itself is already filed as check-skill.sh --require-evals spins forever when its scratch directory vanishes #3682 (--require-evals spins forever when its scratch directory vanishes), so no new issue.
5. ADU.exe reset to 454 handles; it will climb again. Optional, unchanged.

Bisection script (run in an ELEVATED PowerShell; about 1 minute per candidate plus 2)

Samples Token-object growth for 60 s with everything running, then stops one candidate service at a time, re-samples, and restarts it before moving on. The candidate whose stop drops the rate to about zero is the minter. Growth that survives every stop points at a Windows component (CloudAP on this Entra-joined account is the next suspect). Wispr Flow is a user process, not a service: close it by hand and re-sample separately.

Add-Type -TypeDefinition @'
using System; using System.Runtime.InteropServices;
public static class ObjTypes { [DllImport("ntdll.dll")] static extern int NtQueryObject(IntPtr h, int cls, IntPtr buf, int len, out int ret);
  public static long Count(string type) { int len=1024*1024; IntPtr buf=Marshal.AllocHGlobal(len); int ret; if (NtQueryObject(IntPtr.Zero,3,buf,len,out ret)!=0) return -1;
    int n=Marshal.ReadInt32(buf); IntPtr p=buf+8; long r=-1;
    for (int i=0;i<n;i++){ ushort nl=(ushort)Marshal.ReadInt16(p,0); ushort nm=(ushort)Marshal.ReadInt16(p,2); string name=Marshal.PtrToStringUni(Marshal.ReadIntPtr(p,8),nl/2); if(name==type){ r=(uint)Marshal.ReadInt32(p,16); break;} p=p+(0x68+((nm+7)&~7)); }
    Marshal.FreeHGlobal(buf); return r; } }
'@
function Measure-TokenRate([int]$Seconds = 60) {
  $a = [ObjTypes]::Count('Token'); $t = [Diagnostics.Stopwatch]::StartNew(); Start-Sleep $Seconds
  $b = [ObjTypes]::Count('Token'); [math]::Round(($b - $a) / $t.Elapsed.TotalSeconds, 2)
}
$candidates = 'ArmouryCrateService', 'LightingService', 'ROG Live Service', 'AsusFanControlService',
              'AsusUpdateCheck', 'asComSvc', 'AsusCertService',
              'Razer Chroma SDK Service', 'Razer Chroma SDK Server', 'Razer Chroma SDK Diagnostic Service',
              'Razer Game Manager Service 3', 'NvContainerLocalSystem', 'NVDisplay.ContainerLocalSystem'
"baseline (all running): $(Measure-TokenRate) tokens/s"
foreach ($name in $candidates) {
  $svc = Get-Service -Name $name -ErrorAction SilentlyContinue
  if (-not $svc -or $svc.Status -ne 'Running') { "skip: $name ($(if ($svc) { $svc.Status } else { 'absent' }))"; continue }
  try { Stop-Service -Name $name -Force -ErrorAction Stop } catch { "could not stop: $name ($_)"; continue }
  $rate = Measure-TokenRate
  Start-Service -Name $name -ErrorAction SilentlyContinue
  "$name stopped: $rate tokens/s"
}
"all restarted: $(Measure-TokenRate) tokens/s"

What landed in the plugin

#3735 (claude-ops 0.42.0) adds a read-only kernel_objects census to the audit-performance engine (Token objects against uptime, paged pool, findings token-objects-leaked and paged-pool-high), and known-performance-issues.md now carries this signature, the ruled-out list, and the attribution runbook above. The next /claude-ops:audit-performance run on a host in this state names the floor instead of clearing the four suspects and stopping. That PR closes this issue on merge; step 3 stays an operator follow-up, and a recurrence surfaces as a fresh finding in the next audit rather than as a re-derived diagnosis.