chore(rate-limit-guard): commit the #2521 statusline benchmark harness (0.7.4)

plugins/rate-limit-guard/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "rate-limit-guard",
-  "version": "0.7.3",
+  "version": "0.7.4",
   "description": "Shared rate-limit guard for loop lanes: a statusline wrapper tees the subscription rate-limit windows to a fixed machine-scope file, a StopFailure hook records rate-limit stops reactively, and a reader contract fixes how consuming sessions pause and resume.",
   "author": {
     "name": "Melodic Software",

plugins/rate-limit-guard/CHANGELOG.md

@@ -3,6 +3,20 @@
 All notable changes to the `rate-limit-guard` plugin are documented here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning.
 
+## [0.7.4]
+
+### Added
+
+- **bench:** commit the benchmark harness that produced #2521's render-path measurements
+  (`bench/bench-idle.sh`, `bench/bench-load.sh`, `bench/trace-probe.sh`, `bench/lib-bench.sh`),
+  adapted to run from a clean checkout against the repo's own tee, with `bench/README.md`
+  recording the baseline numbers, platform, and spawn-floor method, and `bench/bench.test.sh`
+  smoke-testing the harness in CI — behaviour and output shape only, never timing (#2582).
+  Review hardening over the scratch originals: fork-free timer reads (`printf -v`, no command
+  substitution), a loud bash >= 5.0 refusal instead of an `EPOCHREALTIME` unbound-variable
+  abort, render failures abort a lane instead of being timed, and the load lane's
+  pad-to-one-second arithmetic no longer sleeps 0.1 s when a render took 0 ms.
+
 ## [0.7.3]
 
 ### Changed

plugins/rate-limit-guard/bench/README.md

@@ -0,0 +1,91 @@
+# Statusline render benchmarks
+
+The harness behind the measurements in
+[PR #2521](https://github.com/melodic-software/claude-code-plugins/pull/2521)
+(`perf(rate-limit-guard): spool the statusline snapshot, drain it on a cadence`, 0.7.0).
+It exists so that claim stays reproducible: re-run these lanes before and after any change to the
+statusline tee's render path, and compare against the recorded baseline.
+
+## What #2521 measured
+
+Same-window, Windows 11 / MSYS (Git Bash), n=9 sequential renders:
+
+| configuration | per render |
+| --- | --- |
+| operator's `render.sh` alone | **234.4 ms** |
+| the same `render.sh` behind the pre-#2521 tee | **1047.1 ms** |
+
+The wrapper dominated, and the dominant term inside it was **process creation** — a cost MSYS has
+no cheap primitive for. #2521 removed the forks from the per-render path: the tee now writes the
+session's payload to a per-session spool file with bash builtins only, and one elected session
+drains the spool into the contract snapshot on a cadence (`RLG_TEE_DRAIN_INTERVAL`, default 30 s).
+`trace-probe.sh` is the check that the non-elected render path has stayed that way.
+
+Absolute numbers are machine- and window-specific; the thing to hold onto across runs is the
+**delta between the two configurations**, bracketed by the spawn floor below.
+
+**Instrument note.** The #2521 figures above were taken with the original scratch harness, whose
+timer reads ran as `$(command substitution)` subshells — adding roughly one process spawn *inside*
+each sample. The committed harness reads the clock with `printf -v` (no fork), so it will report
+lower absolute numbers for the same target. The bias was common to both rows of the table, so the
+delta — the claim #2521 rests on — is unaffected; treat the recorded absolutes as
+instrument-inclusive historical values, and re-baseline with the committed harness before using
+absolute numbers in a new claim.
+
+## Method: the spawn floor
+
+On MSYS every number here is dominated by the cost of creating a process, and that cost drifts
+with machine load. Each lane therefore measures the *spawn floor* — the median of 11 bare
+`bash -c exit` spawns (`BENCH_FLOOR_N` overrides the count) — **before and after** the timed
+section, and prints both. A run whose floor moved materially between the two brackets is not
+comparable to its neighbour: discard it. Compare medians, not means; both are printed.
+
+**Bash floor.** The harness requires **bash >= 5.0** and refuses loudly below it. The tee itself
+runs down to bash 3.2 (below 4.2 it degrades to its synchronous path — see the "BASH FLOOR" note
+in `../scripts/statusline-tee.sh`), but the harness is a measuring instrument whose subject is
+process-spawn cost: `EPOCHREALTIME` is the only fork-free clock bash offers, and any fallback
+(`date +%s%3N`) would put a spawn inside every timer read. A failing render likewise aborts the
+lane — a mistyped `STATUSLINE_ENTRY` must never produce plausible-looking numbers.
+
+## Lanes
+
+Runnable from a clean checkout — by default every render invokes this repo's
+`../scripts/statusline-tee.sh` in standalone mode (no wrapped statusline). To measure your real
+statusline path, point `STATUSLINE_ENTRY` at the entrypoint your `settings.json` runs (for
+the #2521 comparison: once at your render script alone, once at the shim/tee wrapping it).
+
+```shell
+# N sequential renders from one idle session (default 11)
+bash plugins/rate-limit-guard/bench/bench-idle.sh 9
+
+# SESSIONS concurrent virtual sessions, one render per second for SECONDS (defaults 10, 60)
+bash plugins/rate-limit-guard/bench/bench-load.sh 10 60
+
+# xtrace of a non-elected render: everything executed before passthrough
+bash plugins/rate-limit-guard/bench/trace-probe.sh
+
+# measuring a machine-local entrypoint instead
+STATUSLINE_ENTRY="$HOME/.claude/statusline/entrypoint.sh" bash plugins/rate-limit-guard/bench/bench-idle.sh
+```
+
+**Isolation:** when a lane exercises the tee, the tee behaves as in production — it spools
+per-session records and (in the elected session) drains them into the machine-scope contract file
+`~/.claude/rate-limit-guard/rate-limits.json`. On a machine whose loop lanes consume that file,
+run the bench against a throwaway HOME so fake `bench-*` sessions never reach real readers:
+
+```shell
+HOME="$(mktemp -d)" bash plugins/rate-limit-guard/bench/bench-idle.sh
+```
+
+(`trace-probe.sh` always isolates itself this way.)
+
+## CI
+
+The **benchmarks gate nothing**: wall-clock numbers on shared CI runners are noise, so no lane's
+timing ever runs in CI. What does run is `bench.test.sh`, a contract smoke suite discovered by
+`scripts/run-plugin-tests.sh` like every other `*.test.sh`: it unit-tests the lib helpers and
+runs each lane once with tiny parameters against the repo tee under an isolated `HOME`, asserting
+behaviour and output shape — never timing. That keeps the harness runnable from a clean checkout
+(an unrunnable harness is exactly the defect that made #2521's measurements unreproducible) and
+maps these files into `scripts/affected-tests.sh` coverage. The tee's behavioural coverage lives
+in `../scripts/statusline-tee.test.sh`.

plugins/rate-limit-guard/bench/bench-idle.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# bench-idle.sh N — N sequential statusline renders, back to back (the shape a
+# single idle session produces). Reports the spawn floor, then per-render ms.
+set -uo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=lib-bench.sh
+source "$DIR/lib-bench.sh"
+
+N="${1:-11}"
+FLOOR_BEFORE="$(spawn_floor)"
+declare -a T=()
+t0=0
+t1=0
+for ((i = 0; i < N; i++)); do
+  now_ms t0
+  if ! render_once "$BENCH_PAYLOAD"; then
+    echo "bench-idle: render failed (STATUSLINE_ENTRY=$STATUSLINE_ENTRY); discarding run" >&2
+    exit 1
+  fi
+  now_ms t1
+  T+=("$((t1 - t0))")
+done
+FLOOR_AFTER="$(spawn_floor)"
+MED="$(printf '%s\n' "${T[@]}" | median)"
+SUM=0
+for x in "${T[@]}"; do SUM=$((SUM + x)); done
+printf 'idle n=%s floor_before=%s floor_after=%s median=%s mean=%s samples=%s\n' \
+  "$N" "$FLOOR_BEFORE" "$FLOOR_AFTER" "$MED" "$((SUM / N))" "${T[*]}"

plugins/rate-limit-guard/bench/bench-load.sh

@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# bench-load.sh SESSIONS SECONDS — SESSIONS concurrent virtual sessions, each
+# firing a statusline render once a second for SECONDS. Reports the spawn floor
+# BEFORE and AFTER (discard the run if they disagree materially), total renders,
+# and the mean render latency across all of them.
+set -uo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=lib-bench.sh
+source "$DIR/lib-bench.sh"
+
+SESSIONS="${1:-10}"
+SECS="${2:-60}"
+OUT="$(mktemp -d)"
+trap 'rm -rf "$OUT"' EXIT
+FAIL_MARKER="$OUT/render-failed"
+
+FLOOR_BEFORE="$(spawn_floor)"
+
+session_worker() {
+  local id="$1" deadline="$2" log="$3" t0 t1 now spent
+  local payload="${BENCH_PAYLOAD/bench-0000/bench-$id}"
+  while
+    now_ms now
+    ((now < deadline))
+  do
+    now_ms t0
+    if ! render_once "$payload"; then
+      : >"$FAIL_MARKER"
+      return 1
+    fi
+    now_ms t1
+    printf '%s\n' "$((t1 - t0))" >>"$log"
+    spent=$((t1 - t0))
+    ((spent < 1000)) && sleep "$(pace_sleep_arg "$spent")"
+  done
+}
+
+START=0
+now_ms START
+DEADLINE=$((START + SECS * 1000))
+for ((s = 0; s < SESSIONS; s++)); do
+  session_worker "$s" "$DEADLINE" "$OUT/s$s.txt" &
+done
+wait
+
+if [[ -e "$FAIL_MARKER" ]]; then
+  echo "bench-load: a render failed (STATUSLINE_ENTRY=$STATUSLINE_ENTRY); discarding run" >&2
+  exit 1
+fi
+
+FLOOR_AFTER="$(spawn_floor)"
+
+TOTAL=0
+SUM=0
+while read -r x; do
+  [[ -n "$x" ]] || continue
+  TOTAL=$((TOTAL + 1))
+  SUM=$((SUM + x))
+done < <(cat "$OUT"/s*.txt 2>/dev/null)
+MED="$(cat "$OUT"/s*.txt 2>/dev/null | median)"
+printf 'load sessions=%s secs=%s floor_before=%s floor_after=%s renders=%s median=%s mean=%s\n' \
+  "$SESSIONS" "$SECS" "$FLOOR_BEFORE" "$FLOOR_AFTER" "$TOTAL" "$MED" \
+  "$((TOTAL > 0 ? SUM / TOTAL : 0))"

plugins/rate-limit-guard/bench/bench.test.sh

@@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+# Contract smoke test for the bench harness: lib-bench.sh helpers, plus one
+# tiny-parameter run of each lane — bench-idle.sh, bench-load.sh, and
+# trace-probe.sh — against the repo's own statusline-tee.sh under an isolated
+# HOME. Asserts behaviour and output shape, never timing: the benchmarks
+# themselves are not CI material (wall-clock numbers on shared runners are
+# noise), but the harness must keep RUNNING from a clean checkout, because an
+# unrunnable harness is exactly the defect that let #2521's measurements go
+# unreproducible (#2582).
+#
+# Self-contained: defines its own assertion helpers — installed plugins are
+# cache-isolated with no shared test lib.
+
+set -uo pipefail
+
+BENCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+LIB="$BENCH_DIR/lib-bench.sh"
+
+PASS=0
+FAIL=0
+fail() {
+  echo "FAIL: $*" >&2
+  FAIL=$((FAIL + 1))
+}
+ok() {
+  echo "ok: $*"
+  PASS=$((PASS + 1))
+}
+
+WORK="$(mktemp -d)"
+cleanup() { rm -rf "$WORK"; }
+trap cleanup EXIT
+
+# Small spawn floors everywhere: the floor's VALUE is irrelevant here, only
+# that the lanes run end to end.
+export BENCH_FLOOR_N=2
+
+# --- lib: median -------------------------------------------------------------
+# shellcheck source=lib-bench.sh
+source "$LIB"
+
+# expect_eq LABEL ACTUAL EXPECTED
+expect_eq() {
+  if [[ "$2" == "$3" ]]; then
+    ok "$1"
+  else
+    fail "$1: got '$2', want '$3'"
+  fi
+}
+
+expect_eq "median: odd count picks the middle" "$(printf '5\n1\n3\n' | median)" "3"
+expect_eq "median: even count picks the upper middle" "$(printf '1\n2\n3\n4\n' | median)" "3"
+expect_eq "median: empty input yields 0" "$(printf '' | median)" "0"
+
+# --- lib: pace_sleep_arg (regression: 0 spent must pad a FULL second; the old
+# --- inline "0.%03d" printed 1000 ms as "0.1000" = 0.1 s) --------------------
+expect_eq "pace_sleep_arg: 0 ms spent pads a full second" "$(pace_sleep_arg 0)" "1.000"
+expect_eq "pace_sleep_arg: 999 ms spent pads 1 ms" "$(pace_sleep_arg 999)" "0.001"
+expect_eq "pace_sleep_arg: 600 ms spent pads 400 ms" "$(pace_sleep_arg 600)" "0.400"
+
+# --- lib: now_ms assigns without printing, monotonically ---------------------
+A=0
+B=0
+now_ms A
+now_ms B
+if [[ "$A" =~ ^[0-9]+$ && "$B" =~ ^[0-9]+$ ]] && ((B >= A)); then
+  ok "now_ms: integer ms, monotone across two reads"
+else
+  fail "now_ms: A=$A B=$B"
+fi
+
+# --- lib: refuses bash without EPOCHREALTIME loudly --------------------------
+OUT="$(bash -c 'unset EPOCHREALTIME; source "$1"' _ "$LIB" 2>&1)"
+RC=$?
+if [[ $RC -ne 0 && "$OUT" == *"requires bash >= 5.0"* ]]; then
+  ok "lib: missing EPOCHREALTIME is a loud refusal, not an unbound-variable abort"
+else
+  fail "lib: EPOCHREALTIME guard rc=$RC out=$OUT"
+fi
+
+# --- bench-idle: smoke run against the repo tee, isolated HOME ---------------
+HOME1="$WORK/home1"
+mkdir -p "$HOME1"
+OUT="$(HOME="$HOME1" bash "$BENCH_DIR/bench-idle.sh" 2 2>&1)"
+RC=$?
+if [[ $RC -eq 0 && "$OUT" =~ ^idle\ n=2\ floor_before=[0-9]+\ floor_after=[0-9]+\ median=[0-9]+\ mean=[0-9]+\ samples=[0-9]+\ [0-9]+$ ]]; then
+  ok "bench-idle: clean-checkout smoke run emits the report line"
+else
+  fail "bench-idle: smoke rc=$RC out=$OUT"
+fi
+
+# --- bench-idle: a failing render aborts the run, never becomes a sample -----
+BAD="$WORK/bad-entry.sh"
+printf '#!/usr/bin/env bash\nexit 7\n' >"$BAD"
+OUT="$(HOME="$HOME1" STATUSLINE_ENTRY="$BAD" bash "$BENCH_DIR/bench-idle.sh" 2 2>&1)"
+RC=$?
+if [[ $RC -ne 0 && "$OUT" == *"render failed"* ]]; then
+  ok "bench-idle: failing render aborts instead of reporting numbers"
+else
+  fail "bench-idle: failing render rc=$RC out=$OUT"
+fi
+
+# --- bench-load: smoke run, 1 virtual session for 1 second -------------------
+HOME2="$WORK/home2"
+mkdir -p "$HOME2"
+OUT="$(HOME="$HOME2" bash "$BENCH_DIR/bench-load.sh" 1 1 2>&1)"
+RC=$?
+if [[ $RC -eq 0 && "$OUT" =~ ^load\ sessions=1\ secs=1\ floor_before=[0-9]+\ floor_after=[0-9]+\ renders=([0-9]+)\ median=[0-9]+\ mean=[0-9]+$ ]] &&
+  ((BASH_REMATCH[1] >= 1)); then
+  ok "bench-load: clean-checkout smoke run emits the report line with >=1 render"
+else
+  fail "bench-load: smoke rc=$RC out=$OUT"
+fi
+
+# --- bench-load: a failing render aborts the run -----------------------------
+OUT="$(HOME="$HOME2" STATUSLINE_ENTRY="$BAD" bash "$BENCH_DIR/bench-load.sh" 1 1 2>&1)"
+RC=$?
+if [[ $RC -ne 0 && "$OUT" == *"render failed"* ]]; then
+  ok "bench-load: failing render aborts instead of reporting numbers"
+else
+  fail "bench-load: failing render rc=$RC out=$OUT"
+fi
+
+# --- trace-probe: prints the pre-passthrough trace of the repo tee -----------
+OUT="$(bash "$BENCH_DIR/trace-probe.sh" 2>&1)"
+RC=$?
+if [[ $RC -eq 0 && "$OUT" == *"=== pre-passthrough trace ==="* && "$OUT" == *"TRACE PATH:"* ]]; then
+  ok "trace-probe: emits the pre-passthrough trace against the repo tee"
+else
+  fail "trace-probe: rc=$RC out=${OUT:0:400}"
+fi
+
+echo
+echo "PASS=$PASS FAIL=$FAIL"
+[[ $FAIL -eq 0 ]]