[inureyes]

Summary

The Activity performance gate judged five alternating pairs by a bare median against the 2% budget and also failed any mode whose baseline CV exceeded 5%. On lablup-dgxspark21 the per-pair noise (SD 3.6 points, kurtosis 12 to 25) makes both rules fire on unchanged code. Each mode now samples until an interval decides, and a mode that cannot be decided fails as noise, not as a slowdown.

The rule

• Per pair: (off - on) / off * 100, alternating order as before. Point estimate: the median. Interval: the distribution-free sign-test interval [x(k), x(n+1-k)] at 1% per tail (98% two-sided). It needs no variance estimate and no random bootstrap, so JavaScript and Python agree bit for bit.
• investigate (slowdown): the interval lies wholly above the 2% budget.
• within-target: it does not, and its upper bound is at most 4%.
• Otherwise add 2 pairs, from 12 up to 40; still undecided at 40 is inconclusive, which fails with a message naming measurement noise.
• Baseline CV and the paired range stay in the summary as diagnostics and decide nothing. The 2% budget is unchanged.
• The 4% bound is the precision a pass must reach, not a looser budget: a 4% bound gives 0.15% null inconclusive per mode, while 3% gives 2.4%, which breaks the 1% false-failure target (table below).
• All constants live in webui/scripts/activity-performance-policy.json. The verifier re-derives every mode from the raw samples with a Python port (scripts/webui/activity_performance_verdict.py) and derives the sample and preflight counts from paired_runs. The summarizer has no samples, so it checks each summary's internal consistency under the same policy. Both languages are tested against webui/scripts/activity-performance-fixtures.json.

Resampling estimate

Noise model: all 636 complete diagnostics artifacts on this runner (2026-09-15 to 2026-10-22, 6,360 pairs), centred on a true 0% change, iid pairs drawn from that pool plus a per-run shift N(0, 0.83^2). 0.83 is the between-run SD estimated from the post-#2119 runs; one-visible and two-visible means in the same run are uncorrelated (r = -0.07). 20,000 simulated runs per row, sequential looks included:

True degradation	investigate	inconclusive	within-target	mean pairs
0%	0.06%	0.15%	99.79%	13.5
2%	11.4%	7.0%	81.6%	20.4
4%	78.6%	8.5%	12.9%	21.6
5%	96.0%	2.3%	1.6%	17.1
6%	99.67%	0.27%	0.07%	14.1
8%	100%	0%	0%	12.4

• False failure on a true 0% change: 0.21% per mode, about 0.4% per CI job (two modes), about 0.6% in full mode. With iid pairs and no run shift it is below 0.01%.
• Detection: at least 95% for true effects of 5% and above, 99.7% at 6%. At 4% it is only 79%, so the injected slowdown for acceptance is sized at about 6%.
• Rejected alternatives: a 3% upper bound (2.4% null inconclusive); passing only when the interval lies wholly below 2% (far more pairs); mean with a t-interval and Hodges-Lehmann (similar power, but the mean is hurt by the heavy tails and HL assumes a symmetry the data lacks, skew -0.7).
• Assumption: the guarantee holds while run-level shifts stay near the measured 0.8 points. A shift SD of 1.5 gives 2.4% false investigate; keeping such shifts out is the job of the fix(ci): clean up the Activity gate server and score on a quiet host #1949 host-quiet precheck.
• Warmup: there is no first-pair bias (pair 0 mean -0.50 vs -0.48 overall, and no order effect), so no leading pairs are discarded. The existing warmup request runs 1.6% slower than the first baseline, and that request is already excluded.

Wall time

Measured per pair on 153 recent runs: one-visible 7.2 s median (7.7 s max), two-visible 8.4 s (9.2 s max), plus about 34 s before the first pair. In CI (visible-only-headed), the worst case at 40 pairs per mode is about 12 minutes, and the expected null run is about 4 minutes (about 2 today). full mode adds a hidden block, so its worst case is about 20 minutes, inside the 1800 s activity timeout. ci.yml timeouts are unchanged.

Fixture provenance

The fixture copies the per-sample logs of the webui-activity-performance-diagnostics-Linux artifacts, none of which had expired: 35423171315 a1, 35425297407 a1/a2/a3, 35425050365 a1, 35421166203 a1, 35567777700 a1 (#1944), 35588958676 a1 (#1946) and 38265665905 a1/a2 (#2279). At their recorded five pairs, every one of them returns continue (no verdict), never investigate. Their concatenated pairs decide within-target at 12 pairs in both modes. The same pairs with the Activity arm 6% slower decide investigate, and spread-amplified or drifting (CV 12%) variants exercise inconclusive and CV-only cases.

Test plan

• pnpm --dir webui run unit and pnpm --dir webui run lint
• make verify-webui-helper-tests, including the new activity_performance_verdict_tests.py
• python3 scripts/webui/verify_activity_performance_tests.py and summarize_activity_evidence_tests.py
• make verify-versions verify-env-var-docs verify-python-tooling
• The new gate inside the real WebUI installed artifact job on this PR, run 39853154267: one-visible within-target (median -0.15%, 98% interval [-1.83%, 1.25%]) and two-visible within-target (median -0.76%, [-2.64%, 0.47%]), both at 12 pairs, with the verifier and summarizer accepting the evidence; the verifier took 3 min 42 s.

Acceptance runs on lablup-dgxspark21

• Five consecutive green WebUI installed artifact runs of this tree (head a7cc03f): run 39853154267 attempts 1 to 5 (jobs 120084449655, 120097378392, 120110916929, 120116442394, 120121248222).
• Injected slowdown, red with investigate in both modes: run 39870659579 on throwaway draft PR test(ci): #1925 injected Activity slowdown red run (do not merge) #2554 (closed, never merged). one-visible median 4.58%, 98% interval [3.89%, 5.68%]; two-visible median 8.05%, interval [7.11%, 9.62%]; 12 pairs each. The injection sleeps 400 us per decode tick for each open /ui-api/v1/events stream, which the Activity page holds open exactly while visible.
• The true effect was not measured beforehand on a separate harness (a local build on the shared CI host was not permitted); it is characterised only by the gate's own median and interval above, both inside the 4 to 8% target for one-visible.
• The issue's suggested injection does not work on GB10. A per-poll CPU burst of 80 spinning threads (run 39862888632) raised the runnable tasks to 46 to 87 in the visible arms, but decode stayed at 152 tok/s in both arms. A delay keyed on Activity polls (runs 39866915190 and 39868894929) also measured 0%: the instrumented run showed that no operations poll ever landed inside a decode window, because each poll was answered the moment a request finished.

Closes #1925