Skip to content

feat(performance): add the measurement-first optimization plugin - #3561

Merged
kyle-sexton merged 11 commits into
mainfrom
feat/performance-plugin
Sep 2, 2026
Merged

kyle-sexton merged 11 commits into
mainfrom
feat/performance-plugin

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the performance plugin: a measurement-first optimization workflow for an arbitrary target,
built around refusing to report what the data does not support.

Generalized from one end-to-end run of that workflow done by hand against the disk-hygiene
destructive-guard hook (#3523). That session had a competent operator and a strong prompt, and it
still produced five verification harnesses that each returned a confident wrong answer rather than
an error
. Four of the five were checks written specifically to avoid being fooled. That
disproportion is the plugin's whole reason for existing: a workflow that measures without enforcing
harness-integrity rules mostly generates confident numbers, which is worse than generating none.

The design was settled by the /planning:interview #3530 requires. Nine questions answered by the
user, three deferred to planning, all recorded in docs/topics/performance-plugin/PLAN.md and
summarized on the issue.

Fix

Four skills, each naming its successor rather than routing through a hub, the way the planning
pipeline already chains:

Skill Owns
target Ranks candidates by evidence quality (E1 attributed measurement to E4 suspicion). Nothing above E3 means the recommendation is "instrument this first", not a ranked guess.
goal Human-gated always. Computes the irreducible floor before the target is agreed, and stops when the target sits below it.
snapshot Qualifies the host, then captures. Interleaved or duet A/B, counter ranked above duration, and the refusal.
verify Fresh-context re-derivation that does not inherit the implementer's numbers, plus a report that never rounds a miss into a win.

Named snapshot, not measure: Q2 locked "depend + route" on /verification:measure, and two
skills called measure is that routing line failing to route. measure keeps baseline capture,
storage and compare mechanics, and gains one gotcha pointing here for hosts a noise-floor warning
cannot describe.

The shared lib becomes a registered cluster. A cross-plugin runtime import was never available,
since plugins install independently. So lib/spawn_noise.py is carried as a byte-identical copy with
scripts/sync-spawn-noise.sh, a registry entry, and the spawn-noise-sync CI lane, the mechanism
this repo already uses for six clusters. The canonical gains is_measurable() (the refusal verdict)
and percentile_floor() (the 1/(1-p) sample floor).

Two places the research contradicted the issue, and the code follows the evidence

Three claims the literature does not ground, labelled rather than dressed up

  • Sample count. No benchmarking-community figure exists beyond the derivable 1/(1-p) floor. The
    p50/p95-over-20 default is a labelled house convention; only the arithmetic floor is enforced.
  • p95 itself. "Median plus a high-order percentile" is grounded (SRE Book ch. 4), but the
    percentiles that chapter names are the 99th and 99.9th.
  • Counts over wall clock is grounded only for instruction counts. Extending it to process
    spawns is this plugin's own generalization, and it is load-bearing here because spawn count is the
    headline metric and Valgrind does not run on Windows.

Two citation traps the skill bodies avoid on purpose: benchstat is unpaired (it recommends
interleaved collection but analyzes with Mann-Whitney U), and coordinated omission is a
load-generator problem, so citing Tene for a synchronous harness would miscite the field's
best-known source.

Verification

  • All four skills PASS check-skill.sh with 0 errors and 0 warnings.
  • scripts/sync-spawn-noise.test.sh — 7 assertions, passing.
  • plugins/claude-ops/lib/spawn_noise.test.sh — 9 assertions, passing.
  • audit_performance.test.sh — 45 tests, passing, unmodified.
  • scripts/check-lane-coverage.sh --check — all 50 lanes reachable from ci-status.needs, including
    the new one.
  • scripts/check-cross-plugin-source-drift.sh --check — no unregistered or drifted clusters.
  • run-ruff.sh clean on both lib copies; markdownlint clean; no em dashes in any new surface.

The gates were proven to discriminate, not assumed to. This is the plugin's own doctrine applied
to its own code, and it matters because four of the five catalogued harness failures were checks that
exited identically in both arms and reported a confident verdict:

  • The sync gate. Its suite drifts a copy's BIMODAL_SPREAD_RATIO and asserts the clean and
    drifted arms return different verdicts, not merely that each printed its expected string.
  • The two-part bimodal predicate. Deleted the high >= SLOW_SPAWN_FLOOR_MS clause by hand; the
    suite failed with the assertion it was written to produce; restored; confirmed with an empty
    git diff rather than trusting the restore. Done after committing, because harness defect feat(hook-telemetry): marketplace-wide telemetry contract + markdown-formatter producer #5
    in the catalogue was a git checkout -- restore over uncommitted work that destroyed it.
  • The refusal itself. test_a_quiet_host_is_measurable_and_a_contended_one_is_not runs both a
    low-variance and a high-variance host and asserts the verdicts differ, because a refusal that fires
    on every host refuses nothing. The snapshot eval suite carries the same positive/negative pair.

The harnesses

Nine scripts under plugins/performance/scripts/, each with a co-located suite, 200 assertions
total. Ported from the source run's scratch tree, which lived on local disk only and would have died
with that directory.

spawn-census.sh / run-spawn-census.sh use a stable shim dir, closing the defect where a
mktemp -d shim put a fresh path on PATH every run against a PATH-keyed cache, so the census
measured its own randomization and reported "no improvement". ab.sh + summarize.py + ratio.py
interleave the arms and flip order per iteration. differential.py proves behavior over an argv
matrix. discriminate.py consolidates five variants, four of which were broken.

The verifiers found seven real defects between them, all fixed. The two that matter most:

  • discriminate.py scored a check that never ran. With no signal configured the signal is the
    exit code, so any shared non-zero rc reported NOT DISCRIMINATING with an affirmatively false
    explanation. It now splits identical-failing (HARNESS BROKEN, exit 2) from identical-passing
    (NOT DISCRIMINATING, exit 1). The four original harness failures that exited 127 in both arms
    would now be caught rather than reported clean.
  • The sample floor guarded one statistic out of three. Two identical true arms produced
    median_paired_ratio=1.06x beside ratio_of_p50=12.08x, so a reader could quote a 12x speedup
    between true and true. All three are gated now.

Also fixed: a 126 subject censused as spawns=0 exit 0; a spliced row ratio.py accepted that
summarize.py rejects; printf | subject under pipefail fabricating exit 141 intermittently on a
pipe-buffer race; and an os reference with no import on a line no test reached, which is exactly
the "check that never ran" shape that harness exists to detect.

Rule 4 is proven behaviorally, not asserted: the target is committed, the fix applied and left
uncommitted, and the fix is still present after the run. A git checkout -- restore would have
destroyed it, which is what defect #5 in the source catalogue actually did.

Related

Closes #3530. Depends on #3553 (merged), which promoted the lib.

Every acceptance criterion on that issue is met: the manifest validates, all four skills pass
check-skill.sh with zero warnings, the refusal is asserted with both a high-variance and a
low-variance arm shown to differ, the drift-immune counter is ranked above any duration in the
emitted report, the design questions were answered by the user in a /planning:interview
linked from the issue,
and every normative claim carries a source tier with the ungrounded ones labelled as house rules.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-01T11:11:01.037553Z eeceb80 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eeceb808ad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/performance/skills/snapshot/SKILL.md Outdated
Comment thread plugins/performance/skills/snapshot/SKILL.md
Comment thread plugins/performance/lib/spawn_noise.py
@kyle-sexton
kyle-sexton force-pushed the feat/performance-plugin branch from eeceb80 to 4ca4a09 Compare September 1, 2026 14:32
@claude

claude Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 3m 53s —— View job


/review:code-review — code-review lane

Followed plugins/review/skills/code-review/SKILL.md directly (invoking it via the Skill tool errored with no output, so I read the skill body and applied it manually). Scope: files changed in this PR (git diff origin/main...HEAD), against CLAUDE.md/repo conventions. Security is out of scope for this lane per the skill's gotchas.

  • Gather context
  • Review changed skill/lib/CI files
  • Post inline findings
  • Post summary

Findings (posted inline)

  1. snapshot/SKILL.md:118 and goal/SKILL.md:113-114 — both skills state an unconditional dependency on /verification:measure for baseline storage, and plugin.json declares no dependency. This diverges from the repo's own established pattern for identical coupling: verification/skills/measure/SKILL.md:67 gates its reciprocal route with "when the performance plugin is installed", and planning/skills/plan/SKILL.md:126 gates its call to /verification:measure with "if installed... Or measure manually". performance doesn't reciprocate either pattern, so a standalone performance install (which the plugin's own README treats as a real usage mode) has no fallback for target → goal → snapshot.
  2. snapshot/SKILL.md:23-27 — the documented host-qualification step (from spawn_noise import spawn_probe, is_measurable) has no path anchoring to ${CLAUDE_PLUGIN_ROOT}/lib, unlike every other cross-plugin script reference in this repo (plugins/claude-ops/skills/*/SKILL.md all anchor via ${CLAUDE_PLUGIN_ROOT}). Following the step as written raises ModuleNotFoundError.
  3. lib/spawn_noise.py:146-156is_measurable() checks bimodal-spawn-latency before spawn-probe-timed-out. When a probe has both a real bimodal spread and a timeout, the reported reason presents the timeout ceiling as a measured finite max and omits that the tail is unbounded. The verdict doesn't change, but the stated basis does — and this module's whole design point is that the basis is what a reader acts on. The new TestMeasurabilityVerdict cases in test_spawn_noise.py don't cover this combined case either.

Findings 1-2 substantively overlap with Codex's automated review on this PR (same two lines); I independently confirmed both by checking actual repo precedent (grep-ing for a real dependencies field in any plugin.json, and comparing against planning:plan's and verification:measure's own gating conventions) rather than taking the automated finding at face value. Finding 3 is additional, not previously flagged.

Everything else checked and clean

  • plugins/performance/lib/spawn_noise.py is byte-identical to the canonical plugins/claude-ops/lib/spawn_noise.py copy (confirmed with diff), matching the PR's sync-cluster claim.
  • scripts/sync-spawn-noise.sh, the CI spawn-noise-sync job, and the cross-plugin-source-registry.txt entry follow the existing sync-cluster pattern used by the other five clusters.
  • target/SKILL.md and verify/SKILL.md — no issues found; evidence-tier and fresh-context-verification discipline is coherent and well-grounded.
  • reference/harness-integrity.md and skills/snapshot/evals/evals.json — solid, and the eval set includes the negative/positive discrimination pair the PR body describes.
  • marketplace.json / docs/CATALOG.md / version bumps on claude-ops and verification — consistent with repo conventions.
  • I did not run check-skill.sh, run-ruff.sh, or the test suites myself (sandboxed Bash required approval I didn't have in this run) — the PR body's verification claims (0 errors/warnings, passing test counts) are author-claimed and unverified by me, not independently confirmed.

Not evaluated: PLAN.md content, README.md/CHANGELOG.md prose, and the citation/grounding claims in the SKILL.md bodies (Duet Benchmarking, SRE Book, coordinated omission) — these are outside the code-review lane's high-signal bar (no correctness bug attaches to them) and security is scoped to the separate security-review lane.

@claude

claude Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 1m 24s —— View job


Claude finished the security review

  • Gather context (PR body, diff, review comments)
  • Run /review:security-review per custom instructions
  • Post findings

CI security review (/review:security-review)

Reviewed the full diff (origin/main...HEAD, 24 files) for logic, trust-boundary, and GitHub Actions security issues (static-analysis-adjacent findings like unpinned actions or pull_request_target misuse are zizmor's lane, not this one).

No security issues found.

What I checked:

  • New spawn-noise-sync CI job (.github/workflows/ci.yml): checkout is SHA-pinned, persist-credentials: false is set, and the job runs no pull_request_target / workflow_run with secrets over untrusted code. No permission widening.
  • plugins/performance/lib/spawn_noise.py and its canonical twin in claude-ops: subprocess.run is called with a fixed argv list (["/bin/sh", "-c", "exit 0"] / ["cmd", "/c", "exit"]), never shell=True, and never with any value derived from user/host input — no command-injection surface. The module's own docstring makes this an explicit design constraint ("Never runs a discovered hook, statusline command, or MCP server").
  • scripts/sync-spawn-noise.sh and scripts/sync-spawn-noise.test.sh: follow the existing sync-cluster.sh pattern used by the other five clusters; no unsanitized interpolation into shell commands, test fixtures build in mktemp -d temp trees.
  • Manifest/registry changes (marketplace.json, plugin.json files, cross-plugin-source-registry.txt): metadata only, no allowed-tools or permission grants added on any of the four new skills (target, goal, snapshot, verify all ship with no allowed-tools frontmatter).
  • New SKILL.md / reference docs: no network calls, eval/exec, or shell-command construction from external input.

Two non-security review comments already posted by Codex on this PR (/verification:measure dependency declaration on snapshot, and the plugin-root import path) are correctness/packaging concerns, not security findings, so they're out of scope for this lane.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Last security-reviewed head: aaa5e718e3269a947b110bd10ce0ce389a4c96d4. On the next push, the relevance gate compares only the commits since this SHA; delete this comment to force a full re-review.

Comment thread plugins/performance/skills/snapshot/SKILL.md Outdated
Comment thread plugins/performance/skills/snapshot/SKILL.md
Comment thread plugins/performance/lib/spawn_noise.py Outdated
Comment thread plugins/performance/skills/goal/SKILL.md Outdated
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Claude has reviewed this PR 5 times. The lane skips further automatic reviews after 5; deleting this comment resets the count.

@claude

claude Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 3m 15s —— View job


/review:code-review — code-review lane

  • Gather context (PR body, diff, prior reviews)
  • Determine review scope: 3 new commits since the last code-review pass (898309b, e40c23d, 94f9db7) added the entire plugins/performance/scripts/ harness directory (~2,500 lines), which has not been reviewed yet
  • Review new scripts/ harness content
  • Re-check status of previously-flagged findings (dependency gating, import anchoring, bimodal/timeout ordering)
  • Post inline findings
  • Post summary

@claude

claude Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 2m 6s —— View job


/review:security-review — security-review lane

  • Gather context (PR diff via gh pr diff, since git diff origin/main...HEAD has no local merge base under this checkout)
  • Invoke /review:security-review (Skill tool errored with no output as it did on the prior run for this PR; read plugins/review/skills/security-review/SKILL.md directly and followed it)
  • Post findings

Result: No security issues found.

This run's HEAD (94f9db7) supersedes the previously security-reviewed commit (4ca4a09) with substantial new content — the entire plugins/performance/scripts/ directory (ab.sh, harness-lib.sh, spawn-census.sh, run-spawn-census.sh, differential.py, discriminate.py, pathfix.py, ratio.py, summarize.py, plus their test files) that wasn't in scope for the prior review. I reviewed that new surface in full, in addition to re-confirming the previously-reviewed files are unchanged in a way that matters here.

What I checked:

  • All subprocess/bash -c call sites in the new scripts (ab.sh, spawn-census.sh, differential.py, discriminate.py, pathfix.py) — every Python subprocess.run call uses an argv list, never shell=True, and every value passed is either a fixed literal or an operator-supplied path/config value, never externally-attacker-controlled input. ab.sh/spawn-census.sh do run bash -c "$CMD_A" etc., but that's the tool's actual purpose (an A/B and spawn-census harness for arbitrary local commands the operator supplies, same trust model as hyperfine/benchstat) — not an injection surface from any untrusted source.
  • discriminate.py's file-patching flow (writes/restores a target file via a byte-verified sidecar, never git checkout --) — no unsafe deserialization, no path traversal beyond operator-supplied config, restore-verification is defense-in-depth for correctness rather than a security boundary.
  • pathfix.py's cygpath invocation — resolved via shutil.which(), invoked as an argv list with a fixed -w flag, no shell.
  • New spawn-noise-sync CI job (.github/workflows/ci.yml) — checkout is SHA-pinned, persist-credentials: false, no pull_request_target/workflow_run with secrets over untrusted code, no permission widening. Matches the existing five sync-cluster jobs' pattern exactly.
  • marketplace.json / .claude/settings.json / plugin manifest changes — metadata and enablement flags only; no allowed-tools or permission grants added on any of the four new skills.
  • plugins/performance/lib/spawn_noise.py — re-confirmed byte-identical to the canonical claude-ops copy; same fixed-argv, no-shell=True design as previously reviewed.

Two non-security findings already posted on this PR (unconditional /verification:measure dependency with no fallback; unanchored spawn_noise import path) are correctness/packaging concerns from the code-review lane, out of scope here.

@kyle-sexton
kyle-sexton force-pushed the feat/performance-plugin branch from 94f9db7 to 20e9433 Compare September 2, 2026 00:35
@claude

claude Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 3m 16s —— View job


/review:code-review — code-review lane

  • Gather context (PR diff, prior reviews, custom instructions)
  • Invoke /review:code-review skill and follow it exactly
  • Confirm status of previously-flagged findings — all three still unresolved at HEAD 20e9433
  • Manually review the new plugins/performance/scripts/** harness content (differential.py, discriminate.py, pathfix.py, ratio.py, summarize.py, ab.sh, harness-lib.sh, spawn-census.sh, run-spawn-census.sh) — found one new correctness issue in differential.py
  • Independent second pass over the same scripts/ surface, adversarial check
  • Post inline findings
  • Post summary

@claude

claude Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 2m 11s —— View job


/review:security-review — security-review lane

  • Gather context (PR body, diff, prior reviews)
  • Invoke /review:security-review per custom instructions
  • Post findings

Result: skip gate hit — no new security-relevant surface since the last full pass

This PR has already had two full security-review passes on this thread: one against the initial skill/lib/manifest surface, and a follow-up that covered the entire plugins/performance/scripts/ harness directory (ab.sh, harness-lib.sh, spawn-census.sh, run-spawn-census.sh, differential.py, discriminate.py, pathfix.py, ratio.py, summarize.py, and their tests) once that directory landed. Both concluded no security issues.

The only commit added since that last full pass is 20e9433, which touches exactly one file:

  • plugins/performance/scripts/ab.test.sh (+10/-1) — swaps a bare "141" substring match for the stricter "141:" rc-census-key match in an existing test assertion, and adds a positive companion assertion that both arms report a clean rc={0: 4} census. Pure test-assertion tightening; no new code path, no subprocess/shell call, no CI workflow or permission change.

Per the skill's skip gate ("this head already has a successful security review that still applies"), there's no new security-relevant surface to review at this head. No security issues found.

Two non-security findings from the earlier code-review pass remain open on this PR (unconditional /verification:measure dependency with no fallback gate; unanchored spawn_noise import path) — those are correctness/packaging concerns from the /review:code-review lane, not security, and are unaffected by this commit.

kyle-sexton and others added 9 commits September 1, 2026 21:18
Records the outcome of the /planning:interview that issue #3530 requires
before implementation. Twelve questions registered, nine answered by the
user, three deferred to planning with arbiter tags.

Two decisions diverge from #3530's own text and say so:

- Phase 4's "suppress the paired ratio under concurrency" is corrected.
  Duet Benchmarking (Bulej et al., ICPE 2020) measured 5.03x and 37.4x
  accuracy improvements from running arms in parallel on shared machines,
  because both arms absorb the same interference. Sequential interleaving
  keeps the suppression rule; simultaneous paired arms do not.
- The unmeasurable-host refusal ships as a house rule, not as field
  consensus. No benchmarking tool surveyed refuses above a variance
  threshold; they warn and print anyway.

The brief also records that the plugin's headline metric, a process-spawn
count, rests on a rationale the literature grounds only for instruction
counts. That gap is labelled rather than smoothed over.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
Q11 — four skills, no router: target, goal, snapshot, verify, each naming
its successor the way the planning pipeline already chains. Named
`snapshot` rather than `measure` because Q2 locked "depend + route" on
/verification:measure and two skills called `measure` is that routing line
failing to route. Harness integrity ships as a shared reference plus a
script rather than a fifth skill; it is a discipline applied inside the
other skills, not a standalone invocation.

Q10 — a cross-plugin runtime import is not available, since plugins
install independently. The interview's "shared lib" answer is implemented
through the mechanism this repo already uses for six other clusters:
canonical source at lib/, byte-identical plugin copies, a dedicated
sync-*.sh gate, a registry entry, and a CI job. One home for the
threshold, loud drift, no runtime coupling.

Recorded while resolving it: the noise threshold is a two-part predicate
(spread ratio >= 3.0 AND max >= the slow-spawn floor), not a bare ratio. A
cold-then-warm spawn pair clears 3x while every sample is still fast, so a
consumer that re-derives a verdict from the ratio alone would report
contention on a healthy host.

Split into two PRs: the lib promotion and claude-ops refactor first, the
new plugin second. The refactor is test-invisible — audit-performance
re-exports the promoted names, so its six existing cases prove it.

Q12 (sample count and percentile choice) stays USER-RESERVED and is
surfaced at the approval gate, not resolved here.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
Partial scaffold. Skills, lib copy, sync gate, and marketplace entry still
to come.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
Implements #3530 to the contract the required /planning:interview settled
(docs/topics/performance-plugin/PLAN.md).

Four skills, each naming its successor rather than routing through a hub:
target ranks candidates by evidence quality (an unmeasured system yields
"instrument this first", not a guess); goal is human-gated and computes the
irreducible floor BEFORE the target is agreed; snapshot qualifies the host
before measuring anything; verify re-derives the result in fresh context and
reports it without rounding a miss into a win.

Named `snapshot`, not `measure`. Q2 of the interview locked "depend + route"
on /verification:measure, and two skills called `measure` is that routing
line failing to route. That skill keeps baseline capture, storage, and the
compare mechanics; this plugin adds what it does not cover and gains a
gotcha pointing here for hosts a noise-floor warning cannot describe.

The lib gains is_measurable() and percentile_floor(), and performance now
carries lib/spawn_noise.py as a registered cross-plugin cluster with a
dedicated sync gate and CI lane, so the bimodal threshold keeps one home.
Plugins install independently, so a runtime import across the boundary was
never available; the byte-identical-copy mechanism this repo already uses
for six clusters is how the constraint is actually met.

Two places where the research contradicted the issue, and the code follows
the evidence:

- #3530 says to suppress the paired ratio under concurrency. Duet
  Benchmarking (Bulej et al., ICPE 2020) measured 5.03x and 37.4x accuracy
  gains from running arms in PARALLEL on shared machines, because both arms
  absorb the same interference. Both modes ship; the suppression rule is
  scoped to the sequential form, and that reconciliation is labelled as this
  plugin's reading rather than a sourced claim.
- The unmeasurable-host refusal ships as a house rule. No surveyed tool
  refuses above a variance threshold; pyperf, Criterion, JMH and benchstat
  all warn and print anyway.

Three claims the literature does not ground are labelled rather than
dressed as consensus: the p50/p95-over-20 sample default (only the derivable
1/(1-p) floor is real, and only that floor is enforced), p95 itself (the SRE
Book names the 99th and 99.9th), and counts-over-wall-clock for anything but
instruction counts, which is load-bearing here because process-spawn count
is the headline metric and Valgrind does not run on Windows.

Verification: all four skills PASS check-skill.sh with zero warnings; the
sync gate's own suite proves --check DISCRIMINATES by asserting the clean
and drifted arms return DIFFERENT verdicts, not merely that each printed its
expected string; ruff clean; markdownlint clean; no em dashes in any new
surface.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
Two CI gates, both real:

plugin-catalog-enablement required an enabledPlugins key. .claude/cloud-bootstrap.sh
computes what it installs from that file, so a catalogued plugin with no key
never loads in a session here.

contract-slice-prune required docs/topics/performance-plugin/ to go. That tree
is contract tier: committed on a task branch, pruned before merge. Its durable
outcomes graduated to issue #3530 first (comment 5501823163) — the Q10/Q11/Q12
resolutions, the two-part bimodal predicate, and why a cross-plugin runtime
import was never available. The interview ledger was already linked there.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
…ites

Completes the plugin. Two parallel workers, each gated by its own
fresh-context verifier that executed rather than read.

## Harnesses (plugins/performance/scripts/)

Nine scripts, each with a co-located test suite, 200 assertions total.
Ported from the source run's scratch tree, which lived on local disk only
and would have died with that directory.

spawn-census.sh / run-spawn-census.sh use a STABLE shim dir, closing the
defect where a mktemp -d shim put a fresh path on PATH every run against a
PATH-keyed cache, so the census measured its own randomization and reported
"no improvement". ab.sh + summarize.py + ratio.py interleave the arms and
flip order per iteration, suppressing the paired ratio under concurrency.
differential.py proves behavior over an argv matrix. discriminate.py
consolidates five variants, four of which were broken.

The verifiers found seven real defects between them, all fixed:

- discriminate.py scored a check that never ran. Any shared non-zero exit
  read as NOT DISCRIMINATING with an affirmatively false explanation. It now
  splits identical-failing (HARNESS BROKEN, exit 2) from identical-passing
  (NOT DISCRIMINATING, exit 1), so the four original harness failures that
  exited 127 in both arms would now be caught rather than reported clean.
- ratio.py printed a headline ratio with no sample floor: two identical arms
  measured 0.78x to 17.12x at five pairs.
- The floor then guarded only the headline. Two identical `true` arms gave
  median_paired_ratio=1.06x beside ratio_of_p50=12.08x, so a reader could
  quote a 12x speedup between `true` and `true`. All three statistics are
  gated now.
- spawn-census.sh censused a 126 subject as spawns=0, exit 0.
- ratio.py accepted a spliced row summarize.py rejects.
- printf | subject under pipefail fabricated exit 141 intermittently on a
  pipe-buffer race whenever the subject did not drain stdin.
- discriminate.py referenced os with no import, on a line no test reached.
  A line no test executes is the exact shape that harness exists to detect.

Rule 4 is proven behaviorally, not asserted: the target is committed, the
fix applied and left uncommitted, and the fix is still present after the run.
A git checkout restore would have destroyed it, which is what defect #5 in
the source catalogue actually did.

## Evals

target (5 cases), goal (6), verify (7), joining snapshot (5). Each case pins
a specific gate whose removal would reintroduce a real failure: E4 suspicion
never outranking E1 measurement, STOP when the target is below the floor,
the differential covering every mode, a miss never rounded into a win.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
The undrained-stdin assertion searched the whole ab.sh output for the
string "141" to prove no fabricated exit code. That collides with timing
data: a legitimate 141ms sample prints min=141ms and fails the assertion
for a reason unrelated to what it tests. An exit code only ever appears as
a dict key, so "141:" is the form that means what was meant.

Adds a positive companion asserting both arms report a clean rc census,
since not-containing a failure string is weaker evidence than containing
the success one.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
…count

The hygiene lane's machine-specific-paths check failed on a comment in
harness-lib.sh that named this machine's actual home directory while
explaining the 8.3 short-name mismatch. The explanation is load-bearing --
a string prefix test between the two spellings finds nothing, so a
temp-rooted shim dir sails through the check meant to catch it -- but it
does not need a real username to make the point.

Also drops "200 assertions across the nine" from the CHANGELOG. The suites
grew after that sentence was written, and a plugin whose whole premise is
refusing to report unsupported numbers should not ship a count that has
already drifted.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
…t p50

The disagreement flag discarded ratio_of_p95 at the call site
(`p95_cell, _ = ...`), so it never inspected the one statistic it printed
without checking. A tail spike in one arm produced
median_paired_ratio=1.00x, ratio_of_p50=1.00x and ratio_of_p95=5.95x on the
same line, with nothing saying the three disagree. A reader quotes whichever
number is printed.

The check now takes the widest spread across all three and names which
statistic diverged. Adds the p95-only divergence case plus its negative arm,
since a flag that fires on every run flags nothing.

Found by the fresh-context verifier, which asked the question the fix for
the earlier ungated-ratio defect did not: gating the subordinates stopped
them printing below the floor, but said nothing about whether they agree
above it.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
@kyle-sexton
kyle-sexton force-pushed the feat/performance-plugin branch from 20e9433 to 7dc4b5b Compare September 2, 2026 01:20
@claude

claude Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 2m 11s —— View job


/review:code-review — code-review lane

  • Gather context (PR diff, prior reviews, custom instructions)
  • Invoke /review:code-review skill and follow it exactly
  • Confirm status of previously-flagged findings at current HEAD 7dc4b5b
  • Review new content since the last code-review pass
  • Post summary (no new inline comments — see below)

Scope of this pass

Two commits landed since the last code-review pass on this PR:

  • c61ffbe — de-identifies a machine-specific home-directory path from a comment (kept the load-bearing 8.3-short-name explanation, generalized to <SHORT~1>/<longname>), and drops a now-stale "200 assertions" count from the CHANGELOG. Comment/prose only, no behavior change.
  • 7dc4b5b — fixes a real defect: the disagreement check only compared median_paired against ratio_of_p50, silently discarding ratio_of_p95 at the call site (p95_cell, _ = ...), so a p95-only divergence (e.g. median_paired_ratio=1.00x ratio_of_p50=1.00x ratio_of_p95=5.95x) printed with no flag. Now takes the widest spread across both subordinate ratios and names whichever one diverged.

I read the fix in full (plugins/performance/scripts/ratio.py:140-171,222-272) and its new test case (ratio.test.sh case 8: a p95-only spike arm plus a negative "three agreeing ratios must not flag" arm). The logic is correct: ratio_of_percentile() already returns None on any of its three refusal gates (pair-count floor, percentile arithmetic floor, zero denominator), and the new loop's min(median_paired, value) <= 0 guard skips those consistently with the prior single-candidate check. Sandboxed Bash in this run requires approval I don't have, so I read the test rather than executing ratio.test.sh myself — the PR body's "passing" claim for this suite is author-claimed and unverified by me here, but the assertions match the fix's stated defect precisely (asserts ratio-of-p95 is named, asserts the paired median and p50 print unmoved, asserts the negative arm prints no DISAGREEMENT). No new findings in either commit.

Status of previously-flagged findings (unchanged, all still open)

None of the above touches the files these are in, so nothing new to add — restating status only, no duplicate inline comments posted this pass:

  1. Unconditional /verification:measure dependency, no fallbacksnapshot/SKILL.md#L117-L118, goal/SKILL.md#L113-L114plugin.json still declares no dependency, and neither skill gates the call the way verification:measure and planning:plan gate this same coupling elsewhere in the repo. Still open.
  2. Unanchored spawn_noise importsnapshot/SKILL.md#L24from spawn_noise import spawn_probe, is_measurable still has no ${CLAUDE_PLUGIN_ROOT}/lib path anchoring; following the mandatory-first host-qualification step as written still raises ModuleNotFoundError. Still open.
  3. bimodal-spawn-latency checked before spawn-probe-timed-outlib/spawn_noise.py#L149-L156 — a probe with both a real bimodal spread and a timeout still reports the timeout ceiling as a measured finite max, omitting the unbounded tail. Verdict unaffected, stated basis still wrong for that combined case. No test in test_spawn_noise.py exercises the combined case. Still open.

@claude

claude Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 1m 57s —— View job


/review:security-review — security-review lane

  • Gather context (PR diff via git diff origin/main...HEAD, prior review history on this thread)
  • Invoke /review:security-review (Skill tool errored with no output, as in prior runs on this PR; read plugins/review/skills/security-review/SKILL.md directly and followed it)
  • Determine scope: commits since the last full security-review pass
  • Review any new/changed surface for security issues
  • Post findings

Result: skip gate hit — no new security-relevant surface since the last full pass

This PR has already had two full security-review passes on this thread (the initial skill/lib/manifest surface, then the entire plugins/performance/scripts/ harness directory once it landed), plus a prior skip-gate pass confirming no new surface as of 20e9433.

Since 20e9433, two more commits landed at this head (7dc4b5b):

  • c61ffbeplugins/performance/scripts/harness-lib.sh (+3/-4): comment-text-only edit, replacing a machine-specific real path/username in an explanatory comment with a placeholder (<SHORT~1>/<longname>). No code change. Also trims a stale assertion count from CHANGELOG.md.
  • 7dc4b5bplugins/performance/scripts/ratio.py (+24/-9) and ratio.test.sh (+23): widens the existing disagreement check to compare ratio_of_p95 against the paired median in addition to ratio_of_p50, so a tail-spike statistic that was previously computed but silently discarded can no longer disagree unflagged. Pure in-process arithmetic on already-collected sample data and stdout formatting — no subprocess, no shell interpolation, no file I/O, no new input surface.

Per the skill's skip gate ("this head already has a successful security review that still applies" / "change has no security-relevant surface after reading the diff"), there is nothing new to review at this head. No security issues found.

Three non-security findings from the code-review lane remain open on this PR (unconditional /verification:measure dependency with no fallback gate; unanchored spawn_noise import path; is_measurable() branch ordering when a probe has both a bimodal spread and a timeout) — those are correctness/packaging concerns from /review:code-review, not security, and are unaffected by these two commits.

… machine path

Three CI failures, all real:

plugin-gate rejected `workflow-stage: discovery` on the target skill. That
value is not in the cheat-sheet generator's enum; the stage for candidate
identification is `explore`. Regenerated docs/SKILL-CHEAT-SHEET.md, which
the same gate compares against the manifests, so all four skills now appear
in it.

machine-specific-paths failed a second time on a comment in
harness-lib.test.sh that the first pass missed: the same 8.3 short-name
explanation, duplicated in the test beside the lib. The explanation is
load-bearing and stays, since the spelling-independence of the temp-root
rejection is the only thing that case proves, but it no longer names a real
home directory.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
@claude

claude Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 1m 11s —— View job


/review:code-review — code-review lane

  • Gather context (PR diff, prior reviews, custom instructions)
  • Invoke /review:code-review skill and follow it exactly
  • Confirm status of previously-flagged findings at current HEAD ff852aa
  • Review new content since the last code-review pass
  • Post summary (no new inline comments — see below)

Scope of this pass

One commit landed since the last code-review pass on this PR (7dc4b5b): ff852aa, touching exactly three files, all CI-failure fixes:

  • plugins/performance/skills/target/SKILL.md#L7workflow-stage: discoveryworkflow-stage: explore. Confirmed against scripts/cheatsheet-config.mjs:14 ({ slug: "explore", heading: "1. Explore", ... }) — explore is the only valid enum value for this stage; discovery was never a member. Correct fix.
  • docs/SKILL-CHEAT-SHEET.md — regenerated to add all four performance skills under the correct stage headings (target under Explore, goal under Plan, snapshot/verify under Verify outcome), consistent with each skill's own workflow-stage metadata. I couldn't re-run node scripts/generate-cheatsheet.mjs --check myself (sandboxed Bash needs approval I don't have in this run), so byte-for-byte generator output is author-claimed and unverified by me, but the added rows are internally consistent with the manifests they summarize.
  • plugins/performance/scripts/harness-lib.test.sh#L77-79 — de-identifies a second machine-specific path (C:\Users\KYLESE~1\...C:\Users\<SHORT~1>\...) from a comment, keeping the load-bearing 8.3-short-name rationale intact. Comment-only, no behavior change.

All three are mechanical, narrowly-scoped fixes that match their stated purpose. No new findings.

Status of previously-flagged findings (unchanged, all still open)

None of the above touches the files these are in:

  1. Unconditional /verification:measure dependency, no fallbacksnapshot/SKILL.md#L117-L118, goal/SKILL.md#L113-L114. plugin.json still declares no dependency; neither skill gates the call the way verification:measure and planning:plan gate this same coupling elsewhere. Still open.
  2. Unanchored spawn_noise importsnapshot/SKILL.md#L24from spawn_noise import spawn_probe, is_measurable still has no ${CLAUDE_PLUGIN_ROOT}/lib path anchoring. Still open.
  3. bimodal-spawn-latency checked before spawn-probe-timed-outlib/spawn_noise.py#L149-L156 — a probe with both a real bimodal spread and a timeout still reports the timeout ceiling as a measured finite max. No test exercises the combined case. Still open.

These three were already posted as inline comments on earlier commits in this thread; not re-posted here to avoid duplication.

@claude

claude Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 43s —— View job


/review:security-review — security-review lane

  • Gather context (PR diff via git log/git show at HEAD ff852aa, prior review history on this thread)
  • Invoke /review:security-review (Skill tool errored with no output, as in prior runs on this PR; read plugins/review/skills/security-review/SKILL.md directly and followed it)
  • Determine scope: commits since the last full security-review pass
  • Review any new/changed surface for security issues
  • Post findings

Result: skip gate hit — no new security-relevant surface since the last full pass

This PR has already had two full security-review passes on this thread (the initial skill/lib/manifest surface, then the entire plugins/performance/scripts/ harness directory once it landed), plus two prior skip-gate passes (at 20e9433 and 7dc4b5b) confirming no new surface.

The only commit added since the last pass (7dc4b5b) is ff852aa, which touches three files, all CI-lint fixups with no security-relevant content:

Per the skill's skip gate ("change has no security-relevant surface after reading the diff" / "this head already has a successful security review that still applies"), there is nothing new to review at this head. No security issues found.

Three non-security findings from the code-review lane remain open on this PR (unconditional /verification:measure dependency with no fallback gate; unanchored spawn_noise import path; is_measurable() branch ordering when a probe has both a bimodal spread and a timeout) — those are correctness/packaging concerns from /review:code-review, not security, and are unaffected by this commit.
·

Both reviewers converged on the same three, independently. All real.

**Timeout now outranks bimodality in is_measurable.** A timed-out sample is
recorded at the timeout ceiling, not as a measurement, so max_ms is censored.
Reporting the bimodal reason first handed the reader a spread computed from
that ceiling and explained it as an observed slow mode, which reads as a
finite measurement of a tail that is actually unbounded. The refusal now
names the timeout count and says max_ms is censored. Covered by a case where
both findings co-occur, which is the only place the precedence is
observable.

**The documented import example did not run.** `from spawn_noise import ...`
raises ModuleNotFoundError unless the caller is already in lib/, because the
lib is plugin-bundled rather than installed. The example now anchors
sys.path to the plugin root and says how to count the parents for a caller
sitting elsewhere. A documented example that fails as written is the same
defect class this plugin exists to prevent, in its own instructions.

**The /verification:measure dependency is presence-gated.** It was written
as an unconditional dependency with no fallback and no manifest declaration.
It is now a preference for reuse: reuse it when the verification plugin is
installed, otherwise capture into the same memory-tier path directly and say
in the report that the capture was unassisted. This plugin's own gates work
either way, and refusing to measure because a sibling plugin is missing
would fail worse than the duplication it avoids. The README says the same
for every neighbour it names, and records that neither plugin imports the
other at runtime since plugins install independently.

Refs #3530

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDkZJ87i7h8J9KzDNruvuS
@claude

claude Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 56s —— View job


/review:security-review — security-review lane

  • Gather context (PR diff via git show aaa5e71, prior review history on this thread)
  • Invoke /review:security-review (Skill tool errored with no output, as in prior runs on this PR; read plugins/review/skills/security-review/SKILL.md directly and followed it)
  • Determine scope: commits since the last full security-review pass
  • Review any new/changed surface for security issues
  • Post findings

Result: skip gate hit — no new security-relevant surface since the last full pass

This PR has already had two full security-review passes on this thread (the initial skill/lib/manifest surface, then the entire plugins/performance/scripts/ harness directory once it landed), plus three prior skip-gate passes (at 20e9433, 7dc4b5b, ff852aa) confirming no new surface each time.

The only commit added since the last pass (ff852aa) is aaa5e71, which resolves all three previously-open code-review findings from this thread. None of the changes touch a security-relevant surface:

Per the skill's skip gate ("change has no security-relevant surface after reading the diff" / "this head already has a successful security review that still applies"), there is nothing new to review at this head. No security issues found.

All three previously-open code-review findings (unconditional /verification:measure dependency with no fallback gate; unanchored spawn_noise import path; is_measurable() branch ordering for combined bimodal+timeout) are addressed by this commit — those were correctness/packaging findings from /review:code-review, not security, so this lane defers to that lane's re-check.

@kyle-sexton

Copy link
Copy Markdown
Contributor Author

All three review findings fixed in aaa5e71. Both reviewers converged on the same three independently, which is a useful signal in itself; all three were real.

1. Timeout now outranks bimodality in is_measurable (P2 / correctness)

A timed-out sample is recorded at the timeout ceiling, not as a measurement, so max_ms is a censored value. Reporting the bimodal reason first handed the reader a spread computed from that ceiling and explained it as an observed slow mode, which reads as a finite measurement of a tail that is actually unbounded.

The check order is now: no-samples, timeout, bimodal, slow-floor. The refusal names the timeout count and says max_ms is censored rather than measured.

Covered by a new case feeding [180.0, 1400.0, 20000.0] with one timeout, which is the only input where both findings co-occur and therefore the only place the precedence is observable. It asserts the reason names the timeout and does not name the bimodal signature, so an inverted precedence fails rather than passing quietly.

2. The documented import example did not run (P1 / correctness)

Correct, and worth stating plainly: a documented example that fails as written is the same defect class this plugin exists to prevent, sitting in its own instructions.

from spawn_noise import ... raises ModuleNotFoundError unless the caller already sits in lib/, because the lib is plugin-bundled rather than installed. The example now anchors sys.path to the plugin root using the same idiom plugins/disk-hygiene/skills/clean/scripts/destructive_guard.py uses, and says to count the parent levels for wherever the caller actually sits rather than copying the index.

3. The /verification:measure dependency is presence-gated (P1 / architecture)

It was written as an unconditional dependency with no gate, no fallback, and no manifest declaration. Now a preference for reuse rather than a requirement: reuse /verification:measure via the Skill tool when the verification plugin is installed, otherwise capture into the same memory-tier path directly and state in the report that the capture was unassisted.

The reasoning, recorded in the README: this plugin's own gates (host qualification, interleaving, the counter, the refusal) work either way, and refusing to measure because a sibling plugin is missing fails worse than the duplication it avoids. The README now says the same for every neighbour it names.

On the manifest-declaration half specifically: no dependency is declared, deliberately. The README states that explicitly rather than leaving it inferable, and also records that neither performance nor claude-ops imports the other at runtime. Plugins install independently, so the shared lib/spawn_noise.py is carried as a byte-identical copy in each, with scripts/sync-spawn-noise.sh and the spawn-noise-sync CI lane keeping the bimodal threshold at exactly one home.

@kyle-sexton
kyle-sexton merged commit b767239 into main Sep 2, 2026
63 checks passed
@kyle-sexton
kyle-sexton deleted the feat/performance-plugin branch September 2, 2026 05:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New plugin: performance — measurement-first optimization workflow (baseline/post snapshots, realistic vs ideal goal tiers, drift-immune metrics)

1 participant