test(desktop-notification): measure C1 fd1-leak invariant differentially - #751
Conversation
The C1 slow-sink detector asserted a fixed `< 2000ms` wall-clock bound against a sink that sleeps to prove the hook does not wait on the backgrounded telemetry sink. On Windows Git Bash the hook's own process-spawn overhead is ~1.6s solo and 4-10s under parallel-suite load, so the fixed bound had a thin-to-negative margin and false-failed with no fd1 leak present. Measure the invariant differentially instead: a fast-sink baseline run captures the machine's current spawn overhead under the same capture, and the slow-sink run's excess over it isolates the leak signal. Overhead cancels, so the check holds under sustained load while still catching a real fd1 leak (the sink's whole sleep lands in the delta). Sink sleep kept small (6s) so it self-expires before the suite's EXIT cleanup rather than locking its stub file on Windows. Co-Authored-By: Claude Opus <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 2m 27s —— View job Security Review — PR #751
ScopeThree files changed: FindingsNo security issues found. The PR is a test-only change. I examined the two new constructs introduced:
epoch_delta_ms() {
local start="$1" end="$2" ss sf es ef
ss="${start%[.,]*}" sf="${start#*[.,]}"
es="${end%[.,]*}" ef="${end#*[.,]}"
echo $(((es * 1000000 + 10#$ef - ss * 1000000 - 10#$sf) / 1000))
}Both call-sites pass Case 13 — differential fd1-leak detector (lines 215–256) SINK_SLEEP=6
BASE_SINK="$(make_sink "cat >/dev/null")"
SLOW_SINK="$(make_sink "cat >/dev/null; sleep $SINK_SLEEP")"
Notes
|
|
Claude finished @kyle-sexton's task in 2m 5s —— View job PR Review —
|
|
Claude finished @kyle-sexton's task in 1m 28s —— View job Security Review — PR #751
ScopeThree files changed: FindingsNo security issues found. I examined the two constructs introduced by this PR:
epoch_delta_ms() {
local start="$1" end="$2" ss sf es ef
ss="${start%[.,]*}" sf="${start#*[.,]}"
es="${end%[.,]*}" ef="${end#*[.,]}"
echo $(((es * 1000000 + 10#$ef - ss * 1000000 - 10#$sf) / 1000))
}Both call-sites pass Case 13 — differential fd1-leak detector (lines 215–256) SINK_SLEEP=6
BASE_SINK="$(make_sink "cat >/dev/null")"
SLOW_SINK="$(make_sink "cat >/dev/null; sleep $SINK_SLEEP")"
Notes
|
|
Claude finished @kyle-sexton's task in 1m 42s —— View job PR Review —
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 806cae0fc9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ication-c1-threshold # Conflicts: # plugins/desktop-notification/CHANGELOG.md
|
Claude finished @kyle-sexton's task in 2m 9s —— View job PR Review —
|
|
Claude finished @kyle-sexton's task in 1m 16s —— View job Security Review — PR #751
ScopeThree files changed: FindingsNo security issues found.
Both call-sites (L239, L246) pass Case 13 — differential fd1-leak detector (lines 231–256) SINK_SLEEP=6
BASE_SINK="$(make_sink "cat >/dev/null")"
SLOW_SINK="$(make_sink "cat >/dev/null; sleep $SINK_SLEEP")"Both new Notes
|
…gative The C1 fd1-leak detector subtracted a single fast-sink baseline from the slow-sink run. Sequential samples run under different scheduling, so one baseline descheduled >THRESHOLD_MS longer than the slow run could shrink DELTA_MS below threshold and let a real fd1 leak PASS — a false-negative, the detector reporting green while a leak exists (fail-open under load). Take the baseline as the minimum of several fast runs: only an inflated baseline can mask a leak, so the minimum reflects true-minimal overhead and keeps the leak signal (~SINK_SLEEP) in the delta. The inverse error (inflated slow sample, no leak) only re-fails a green run (fail-safe), so the slow run stays single. Suite passes 54/0; shellcheck clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HZqwoLTHPpPRzLpRGzbKRq
|
Claude finished @kyle-sexton's task in 1m 42s —— View job Security Review — PR #751
ScopeThis pass reviews the HEAD commit FindingsNo security issues found. I examined the two constructs introduced or modified by New BASE_SAMPLES=3
...
for ((_i = 0; _i < BASE_SAMPLES; _i++)); do
_TS0=$EPOCHREALTIME
_OUT_BASE="$(run "$(build_input permission_prompt)" HOOK_TELEMETRY_SINK="$BASE_SINK")"
_TS1=$EPOCHREALTIME
_b=$(epoch_delta_ms "$_TS0" "$_TS1")
if [[ -z "$BASE_MS" || $_b -lt $BASE_MS ]]; then BASE_MS=$_b; fi
done
Updated SLOW_SINK="$(make_sink "cat >/dev/null; sleep $SINK_SLEEP")"
Updated diagnostic echo " (C1 fd1-leak: base=${BASE_MS}ms (min of ${BASE_SAMPLES}) slow=${SLOW_MS}ms ...)"
Notes
|
|
Claude finished @kyle-sexton's task in 1m 51s —— View job PR Review —
|
…nk invariant, not half the sleep (#1362) Closes #448 ## Summary - The `#751` fix that closed `#448` could not have worked: `THRESHOLD_MS` was derived as `SINK_SLEEP * 1000 / 2`, so widening `SINK_SLEEP` widened the threshold by the same ratio and left the margin unchanged by construction. `#448` was reopened after this reproduced on clean `main` (`delta=3697ms` false-fail with no leak present), and the same construction was found ported verbatim into `markdown-format`'s C1 copy (from `#1209`). - `THRESHOLD_MS` in both plugins' C1 fd1-leak detector now asserts the actual invariant directly — sink-sleep-minus-a-safety-margin (`SAFETY_MARGIN_MS`), not half the sleep — and `SINK_SLEEP` widens from 6s to 8s (still under the 10s ceiling documented against EXIT-cleanup file-locking on Windows) for more absolute separation between ambient noise and the leak signal. - Both constants were sized against measurements taken on this machine: the reported 3697ms false-fail, and up to ~2150ms of noise generated by 30 concurrent full-suite runs under heavy load — the fixed threshold clears both with comfortable margin while staying meaningfully below the ~8000ms leak signal. - Per-plugin version bump + CHANGELOG entry in both `desktop-notification` and `markdown-format`. ## Test plan Windows Git Bash (the platform this flake is specific to — Ubuntu CI spawns ~10x cheaper and was always unaffected). - [x] 10 consecutive clean sequential runs of `desktop-notification.test.sh` — all green (`PASS=54 FAIL=0`), C1 deltas -175ms to 644ms against the new `<6000ms` threshold. - [x] 40 runs under heavy concurrent load (30x `desktop-notification.test.sh` + 10x `markdown-format.test.sh` launched simultaneously) — all green, worst observed no-leak delta ~1590ms (desktop-notification) / ~1555ms (markdown-format), no `FAIL`. - [x] Deliberately reintroduced the fd1 leak (dropped the `>/dev/null` redirect on the sink spawn in `hook::emit_telemetry`, `plugins/desktop-notification/hooks/hook-utils.sh:455`) — C1 correctly went red: `delta 8065ms ≈ sink's 8s sleep`. Reverted (`git diff` against the reverted file is empty — confirmed clean). - [x] `shellcheck` clean on both modified test files. - [x] Full suites green after the fix: `desktop-notification` `PASS=54 FAIL=0`, `markdown-format` `PASS=65 FAIL=0`. ## Related #443 (the `hook::buffer_stdin` migration that thinned the original margin). #751 (the fix that attempted to close #448 but could not — see Summary). Epic #313 (closed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Summary
The
desktop-notification.test.shC1 fd1-leak detector false-failed on Windows Git Bash. It asserted the hook returns in a fixed< 2000msagainst a sink that sleeps 3s — but the hook's own process-spawn overhead (jq/tr/awk subshells) is ~1.6s solo and 4-10s under parallel-suite load, so the bound had a thin-to-negative margin even though no fd1 leak exists. On this dev machine it already failed outright:2023mselapsed, pure overhead, no leak.Fix
Measure the invariant differentially instead of against a fixed wall-clock bound. The real invariant is "the hook did not wait for the backgrounded telemetry sink." A fast-sink baseline run captures the machine's current spawn overhead under the same command-substitution capture; the slow-sink run's excess over that baseline isolates the leak signal. Ambient overhead cancels in the subtraction, so the check holds under sustained load. Under a leak the sink's whole sleep lands in the delta; with no leak the delta is ~0 (± scheduling jitter). Threshold is half the sink sleep — comfortably above observed jitter, comfortably below the leak signal.
Why differential over the issue's literal Option A/B (
< sink_sleep, or a proportional bump): both are still fixed thresholds measured against variable overhead, and the actual trigger is sustained load (parallel suites), under which overhead rises on both the baseline and the measured run together. A differential cancels that; a fixed bound eventually loses. Fixed< sink_sleepwould additionally needsink_sleep > ~10sto clear the observed 4-10s overhead — a sink that then lingers past the suite's EXIT cleanup and locks its stub file on Windows. The differential keeps the sleep small (6s), so it self-expires during the post-C1 cases before cleanup.Verification
Windows Git Bash (Ubuntu CI spawns ~10x cheaper — always had a huge margin and is unaffected).
Before (origin/main, fixed
< 2000ms) — false-fail with no leak:After — 5 back-to-back runs, all green (worst observed base/slow jitter ~0.85s vs 3000ms threshold):
Detector still catches a real leak — temporarily dropped the sink-spawn
>/dev/nullredirect inhook::emit_telemetry(fd1 inherited by the backgrounded sink), C1 went RED, then reverted:Full suite green after the fix:
PASS=54 FAIL=0. No EXIT-cleanup errors across runs (the 6s sink self-expires during the later cases). Only the test file changed behaviorally;hook-utils.shis unmodified (leak simulation was reverted).Closes #448
Related
#443 (the
hook::buffer_stdinmigration that thinned the margin). Epic #313 (closed).