Skip to content

feat(claude-ops): audit-performance kernel_objects census, the host-level floor beneath the four suspects - #3735

Merged
kyle-sexton merged 3 commits into
mainfrom
perf/3715-host-spawn-floor
Sep 5, 2026
Merged

kyle-sexton merged 3 commits into
mainfrom
perf/3715-host-spawn-floor

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Closes #3715

Summary

claude-ops 0.41.14 to 0.42.0. The audit-performance engine gains a read-only, Windows-only kernel_objects census: live and high-water counts for Token, Process, Thread, Key, File, Section, Event, and EtwRegistration objects through NtQueryObject(ObjectTypesInformation), paged and nonpaged pool plus system handle, process, and thread totals through GetPerformanceInfo, and uptime. It reports objects_per_uptime_second, a population ratio that names its own bias in a basis string, projects the hours to a 250,000-object leak threshold at that ratio, and carries two findings that are not one verdict: token-objects-leaked alone yields state_label: token-leak, while paged-pool-high is an unattributed pool signal routed to poolmon. Off Windows it returns supported: false with the reason instead of vanishing. known-performance-issues.md records the signature, the ruled-out causes, and the elevated attribution runbook, and SKILL.md reads the section before the four suspects.

Why: the host behind #3715 carried 3.81M Token objects and about 10 GB of paged pool at two days' uptime, every process creation on it cost 1.4 to 4 s at 7% CPU, and all four existing suspects cleared. A reboot restored a 14 ms floor and the leak re-armed at once (5 to 15 objects per second post-reboot). The audit could not see this floor before; now it names it and routes the reboot and the attribution to the operator. The issue's remaining step, attributing the minter from an elevated shell, is host operations: the post-reboot readings and a paste-ready bisection script are posted on the issue, and a recurrence now surfaces as token-objects-leaked in the next audit instead of as a re-derived diagnosis.

Fix

  • scripts/audit_performance.py: KERNEL_OBJECT_TYPES, TOKEN_LEAK_OBJECTS, PAGED_POOL_HIGH_MB, and the x64 struct offsets; parse_object_types (a pure memory walk, portable to any platform); _windows_object_type_table; _windows_performance_info; summarize_kernel_objects; kernel_objects; wired into main() as the phase after plugin_fleet; the module docstring names the host-level floor.
  • scripts/test_audit_performance.py: TestKernelObjectCensus (synthetic x64 block parse including name padding, leak labelling, pool-alone labelling, fresh-boot projection, zero-uptime guard, off-Windows mock, live contract keys).
  • scripts/audit_performance.test.sh: asserts the section ships and survives --skip-fan-out.
  • lib/test_spawn_noise.py: the stand-alone check now runs in a python -I child that proves the engine's script directory is absent before importing, because CI's Python lane collects this suite and the engine's in one pytest process and pytest prepends each suite's directory (this is what turned test-linux red on the first push).
  • reference/known-performance-issues.md: the new "The host-level floor" section, measurement-method step 6, and a Windows-amplifier bullet.
  • SKILL.md: description clause, a read-kernel_objects-first paragraph ahead of Suspect 1, and a gotcha.
  • .claude-plugin/plugin.json, CHANGELOG.md, README.md: version and description.

Verification

  • python -m unittest -v test_audit_performance: 52 tests OK, 7 of them new. python -m unittest test_spawn_noise: 10 OK.
  • CI's Python lane emulated locally (python -m pytest -q over both suites in one process, pytest 9.1.1): 62 passed.
  • bash audit_performance.test.sh: prints "OK: kernel-object census is present in the shipped report" and confirms --skip-fan-out keeps the section.
  • scripts/affected-tests.sh --run: both shell lanes pass.
  • scripts/run-ruff.sh check on the three Python files: "All checks passed!".
  • scripts/check-changelog-parity.sh --check, --check-order, and --check-bump origin/main: all pass.
  • scripts/check-purged-em-dashes.sh: no em dashes. markdownlint-cli2 on the changed markdown files: 0 issues.
  • plugins/skill-quality/scripts/check-skill.sh audit-performance against origin/main: PASS, 0 errors, 3 warnings of pre-existing classes.
  • Live cross-check on the reporting host: the engine's Token.objects and an independent C# NtQueryObject reader agree within 0.6%, and pool.paged_mb matches the \Memory\Pool Paged Bytes counter within 0.4%.
  • A fresh-context reviewer found nothing blocking. A fresh-context verifier passed all eight acceptance criteria, including "the engine writes no file" (git status --porcelain identical before and after a run).
  • Review findings from Codex (P1 pool-alone verdict, P2 ratio naming) and the claude lane (timeline reconciliation) were each classified VALID and fixed in fab7708; threads carry the evidence.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01D8ExixbqhNTwnH14jLx7Nt

…evel floor beneath the four suspects

A read-only Windows census through NtQueryObject(ObjectTypesInformation)
and GetPerformanceInfo: live and high-water counts for Token, Process,
Thread, Key, File, Section, Event, and EtwRegistration objects, paged and
nonpaged pool, system handle/process/thread totals, and uptime. Token
leads because a Token object that outlives every handle to it is held by
a kernel reference, and a leaking driver or service path accumulates them
for the life of the boot: the host behind this carried 3.81M Token
objects and ~10 GB of paged pool at two days' uptime, every process
creation cost 1.4 to 4 s at 7% CPU, and a reboot restored a 14 ms floor.

The section reports the mint rate as objects / uptime (a 3 s window read
0/s while a 60 s window read 15/s, so a short window under-reads),
projects the hours to the 250,000-object threshold at that rate, and
labels token-objects-leaked / paged-pool-high. Off Windows it says
supported: false with the reason. The block parser is unit-tested on a
synthetic x64 layout on every platform; the e2e contract asserts the
section ships; the live parse was cross-checked against an independent
C# reader within 0.6%.

known-performance-issues.md records the signature, the ruled-out causes,
and the elevated attribution runbook; SKILL.md reads kernel_objects
before the four suspects. claude-ops 0.41.14 -> 0.42.0.

Refs #3715

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8ExixbqhNTwnH14jLx7Nt
@kyle-sexton
kyle-sexton marked this pull request as ready for review September 5, 2026 01:41
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 5, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-05T01:44:34.622811Z 53824e9 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude

claude Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 4m 10s —— View job


Code review complete

  • Gather context (PR diff, CLAUDE.md, skill instructions)
  • Check skip gate (PR open, non-trivial diff, no prior review at this head → proceed)
  • Review changed files for correctness and maintainability
  • Post findings (inline + summary)

Reviewed per /review:code-review (org CI code-review lane): correctness and maintainability only, scoped to files changed in this PR (origin/main...HEAD).

Findings

One inline comment posted on known-performance-issues.md (lines 174–178): the "two hours after reboot" Token-object count in the narrative bullet (37,144 objects, 5.1/s, ~12 h to threshold) contradicts the table two lines above (line 153: 25,623 objects for the same "2 h uptime" checkpoint). The table's figure is the one that's actually load-bearing — it matches both TOKEN_LEAK_OBJECTS's calibration comment in audit_performance.py (lines 133–136) and the unit test's fresh-boot fixture (25,623 objects / 3.65 per second / ~17.1 h) — so the narrative bullet reads as an unreconciled second sample for the same measurement point. Worth a fix since this doc is the calibration basis an operator would check their own host's numbers against.

What held up well

  • parse_object_types's x64 OBJECT_TYPE_INFORMATION offsets (_OTI_TOTAL_OBJECTS=0x10, _OTI_HIGH_WATER_OBJECTS=0x28, entry size 0x68) check out against the public struct layout, and the padding/decoding logic (UTF-16LE by hand rather than wstring_at, to stay portable to non-Windows wchar_t widths) is correct.
  • Division-by-zero and negative-rate edge cases in summarize_kernel_objects (zero uptime, already-past-threshold) are handled correctly and are covered by tests.
  • The non-Windows and 32-bit-interpreter fallbacks in kernel_objects() return a labelled supported: false rather than raising or silently omitting the section, consistent with the rest of the engine's phase-error handling in main().
  • Version bump (0.41.14 → 0.42.0), changelog, README, and plugin.json description are all consistent with a new read-only feature (minor bump).

Not independently verified

Per the review lane's rules, the PR author's own claimed verification (unit test run, ruff, affected-tests.sh, live cross-check against a C# NtQueryObject reader, changelog-parity checks) is noted as author-claimed, not confirmed by this review — this sandbox's Bash access is restricted to git/read-only inspection, so those commands were not re-run here.

@claude

claude Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 1m 12s —— View job


Security review complete

  • Gather context (PR diff, changed files)
  • Check skip gate
  • Review diff for security-relevant surface
  • Adversarial validation of any candidates
  • Post final findings

No security issues found.

Reviewed all 8 changed files (git diff origin/main...HEAD). The change is a read-only, Windows-only diagnostic addition to scripts/audit_performance.py (kernel_objects, parse_object_types, _windows_object_type_table, _windows_performance_info, summarize_kernel_objects) plus tests and docs. Specifically checked for:

  • Injection surfaces: none. No subprocess/shell calls added, no string-built commands, no template rendering.
  • Untrusted input handling: the new code's only inputs are kernel-supplied data via NtQueryObject(ObjectTypesInformation) and GetPerformanceInfo/K32GetPerformanceInfo — both trusted OS syscalls, not attacker- or network-controlled. The raw pointer walk in parse_object_types trusts offsets/lengths embedded in that kernel-returned block, but there's no trust-boundary crossing here (same-process memory, same privilege level as the caller), so this is a robustness concern rather than an exploitable vulnerability.
  • Secrets/credentials/tokens: none touched. "Token" here refers to Windows kernel Token objects (security tokens counted, never read or dumped), not credentials.
  • File writes / mutation: none — the engine remains read-only per its own docstring and the PR's verified git status --porcelain check.
  • GitHub Actions: no workflow files changed in this PR.
  • Docs (known-performance-issues.md, SKILL.md): the attribution runbook only names Windows services to stop/restart via prose for a human operating an elevated shell; it introduces no executable script or automation path.

Nothing here crosses a trust boundary, handles untrusted input, or touches auth/secrets/CI. No findings to report.

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Last security-reviewed head: 53824e93074b8c5d05b2c7c7b9d4fc07a9d02e46. On the next push, the relevance gate compares only the commits since this SHA; delete this comment to force a full re-review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 53824e9307

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/claude-ops/skills/audit-performance/scripts/audit_performance.py Outdated
Comment thread plugins/claude-ops/skills/audit-performance/reference/known-performance-issues.md Outdated
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count.

kyle-sexton and others added 2 commits September 4, 2026 21:51
…nd names its ratio honestly

Review findings on #3735, all classified VALID:

- paged-pool-high alone no longer yields a leak verdict. GetPerformanceInfo
  reports aggregate pool with no attribution, so state_label is now
  `token-leak` only on token-objects-leaked, `paged-pool-high` when only the
  pool finding fires, and SKILL.md routes the latter to poolmon rather than
  to a reboot.
- per_second_since_boot is renamed objects_per_uptime_second and the
  projection hours_to_leak_threshold_at_uptime_ratio; a `basis` string says
  the ratio includes the boot population (overstates early in a boot, so the
  projection errs short) and cannot see churn. The ratio stays instead of an
  in-run delta because a 3 s window read 0/s against a 60 s window's 15/s on
  the reporting host; the 60 s manual sample is the mint-rate measurement.
- The reference's post-reboot timeline no longer presents two samples four
  minutes apart as one checkpoint: the table carries 1 h 57 min (25,623, the
  calibration basis) and the bullet its own 2 h 01 min and 2 h 10 min
  readings.

CI's Python lane collects lib/test_spawn_noise.py and the engine's suite in
one pytest process, and pytest prepends each suite's directory, so the lib's
stand-alone check (engine script dir absent from sys.path) failed there. It
now runs in a `python -I` child that proves the exclusion itself before
importing, so it passes wherever it is collected.

Refs #3715

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8ExixbqhNTwnH14jLx7Nt
…ption

`node scripts/generate-catalog.mjs`, so the catalog block carries the
audit-performance kernel-object census clause the plugin manifest now
declares; test-linux's manifest-and-catalog validation gates on parity.

Refs #3715

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8ExixbqhNTwnH14jLx7Nt
@kyle-sexton
kyle-sexton merged commit 50304c1 into main Sep 5, 2026
23 of 26 checks passed
@kyle-sexton
kyle-sexton deleted the perf/3715-host-spawn-floor branch September 5, 2026 02:06
@kyle-sexton
kyle-sexton restored the perf/3715-host-spawn-floor branch September 15, 2026 18:29
@kyle-sexton
kyle-sexton deleted the perf/3715-host-spawn-floor branch September 19, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(host): melo-desk-001 process creation costs 1.4 to 4 s system-wide; 3.81M leaked kernel Token objects (~10 GB paged pool) since boot

1 participant