diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4613c04a9..87494b712 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -841,6 +841,7 @@ jobs: python3 scripts/webui/verify_generated_key_helper_tests.py python3 scripts/webui/verify_reverse_proxy_tests.py python3 scripts/webui/verify_single_model_tests.py + python3 scripts/webui/activity_performance_verdict_tests.py python3 scripts/webui/verify_activity_performance_tests.py python3 scripts/webui/summarize_activity_evidence_tests.py python3 scripts/webui/activity_gate_host_tests.py diff --git a/Makefile b/Makefile index 64c17e4bd..519c0137a 100644 --- a/Makefile +++ b/Makefile @@ -1000,6 +1000,7 @@ verify-webui-helper-tests: ## Run pure-Python WebUI verifier helper tests; no se @$(WEBUI_BUNDLE_PY) scripts/webui/verify_generated_key_helper_tests.py @$(WEBUI_BUNDLE_PY) scripts/webui/measure_startup_tests.py @$(WEBUI_BUNDLE_PY) scripts/webui/verify_single_model_tests.py + @$(WEBUI_BUNDLE_PY) scripts/webui/activity_performance_verdict_tests.py @$(WEBUI_BUNDLE_PY) scripts/webui/verify_activity_performance_tests.py @$(WEBUI_BUNDLE_PY) scripts/webui/summarize_activity_evidence_tests.py @$(WEBUI_BUNDLE_PY) scripts/webui/activity_gate_host_tests.py diff --git a/TECHNICAL_REPORTS/2553-activity-gate-interval-verdict-20261023.en.md b/TECHNICAL_REPORTS/2553-activity-gate-interval-verdict-20261023.en.md new file mode 100644 index 000000000..7e0fba66b --- /dev/null +++ b/TECHNICAL_REPORTS/2553-activity-gate-interval-verdict-20261023.en.md @@ -0,0 +1,124 @@ +# PR #2553: Judge Activity decode overhead by an interval, not 5 pairs + +**Date**: 2026-10-23 +**Status**: PR open; Closes #1925 + +## Executive Summary + +The `WebUI installed artifact` job's Activity performance gate measured five alternating off/on pairs per mode and failed the mode when the median paired degradation exceeded the 2% budget or the baseline CV exceeded 5%. On the self-hosted GB10 runner `lablup-dgxspark21` the per-pair noise (SD 3.6 points, kurtosis 12 to 25) makes both conditions fire on unchanged code: #1925 and its comments record ten failed attempts on PRs that could not affect decode, including one that failed on CV while the Activity arm measured 6.76% faster. PR #2553 replaces the rule with a sequential, distribution-free test: the point estimate is the median of the paired degradations, the interval is the 98% sign-test interval `[x(k), x(n+1-k)]`, and each mode samples from 12 pairs in steps of 2 up to 40 until the interval decides `investigate` (wholly above 2%), `within-target` (upper bound at most 4%), or, at the cap, `inconclusive`, which fails with a message naming measurement noise. All constants move to one policy file read by the JavaScript harness and a Python port; the verifier re-derives every mode's verdict from the raw samples, and both languages are tested against one shared fixture of recorded sample vectors. A resampling estimate over 636 recorded runs gives 0.21% false failure per mode on a true 0% change and at least 95% detection at 5% and above. On the runner, five consecutive attempts of the unchanged tree passed, and an injected slowdown on a throwaway PR failed with `investigate` in both modes. The issue's suggested injections (a CPU burst, a delay keyed on Activity polls) did not reach decode on GB10 at all. 15 files, +3995/-120, of which 3278 lines are the fixture. + +## 1. Problem Statement + +The gate's verdict lived in `webui/scripts/activity-performance.mjs`: 5 pairs per mode, `status: degradation > 2 || cv > 5 ? 'investigate' : 'within-target'`. The same rule was enforced twice more with independent copies of the thresholds and counts: `scripts/webui/verify_activity_performance.py` required `paired_runs == 5`, re-applied `degradation > 2 or cv > 5`, and hard-coded the per-mode sample and preflight counts; `scripts/webui/summarize_activity_evidence.py` repeated the status, `paired_runs == 5`, `<= 2` and `<= 5` checks. + +Two defects followed from that rule: + +- **Five pairs cannot resolve 2%.** The per-pair spread in the failed runs was 6 to 21 points. Flat changes crossed the median threshold: PR #1920 (display-name strings only) failed one-visible at +3.38%, and PR #2279 (YAML parser only) failed one-visible at +4.65% on its second attempt. +- **CV measured drift, not overhead.** Baseline CV mostly records slow drift across the block, which the alternating pairs exist to cancel. In run 35425050365 attempt 1 the `off` rate fell from 171.4 to 141.7 tok/s over the two-visible block and the mode failed on CV 7.45 although the Activity arm was 6.76% faster. PR #2279 attempt 1 failed on CV 5.17 with every pair faster. + +Each rerun on the single GB10 runner costs 10 to 40 minutes. The issue rejected raising the 2% or 5% thresholds, since that hides noise and real regressions alike. A maintainer comment on the issue added that host-level contention (a concurrent compile on the same machine, invisible to the GPU-process check) also moved scores, for example PR #1944 and #1946; that half was split into #1949 (host-quiet precheck), and this PR covers only the verdict. + +## 2. Change Summary + +| Area | Change | +|---|---| +| `webui/scripts/activity-performance-policy.json` (new) | The only copy of the constants: `budget_percent` 2, `precision_ceiling_percent` 4, `tail_alpha` 0.01, `min_pairs` 12, `pair_step` 2, `max_pairs` 40 | +| `webui/scripts/activity-performance-config.mjs` | Pure verdict functions: `validatePolicy`, `verdictPolicy` (loaded and validated at import), `signTestRank`, `pairDegradation`, `medianInterval`, `lookVerdict`, `sequentialVerdict`, `verdictRule`, `modeSummary`, `pairsFromSamples`; `performanceCompletion` gains `inconclusive` | +| `webui/scripts/activity-performance.mjs` | Sampling loop runs until `lookVerdict` leaves `continue`; prints `mode: status: rule` per mode; writes `verdict_policy` into the output; the old `median` helper (which took the upper middle element for even n) is gone | +| `scripts/webui/activity_performance_verdict.py` (new) | Python port of the same functions, operation for operation, plus `check_summary_consistency`, `failure_message`, `expected_counts` | +| `scripts/webui/verify_activity_performance.py` | Re-derives each mode from `samples`; exact match on `status`, `paired_runs`, median and interval; 1e-9 relative tolerance on CV and paired range; sample and preflight counts derived from `paired_runs`, replacing the two hard-coded count tables | +| `scripts/webui/summarize_activity_evidence.py` | No samples available, so it checks structure and internal consistency under the policy; requires `verdict_policy` to equal the repository's; emits `interval_percent` and `verdict_policy`; CV is no longer bounded above | +| `webui/scripts/activity-performance-fixtures.json` (new) | 22 cases: the ten recorded failing attempts, two concatenated null sequences, synthetic slowdown, speedup, drift, inconclusive and boundary cases, four rejection cases | +| `webui/scripts/activity-performance-verdict.spec.mjs` (new), `scripts/webui/activity_performance_verdict_tests.py` (new) | Both evaluate every fixture case and must reproduce `expected` exactly; JS also tests the sign-test rank, look schedule, policy validation and `performanceCompletion` | +| `webui/scripts/activity-stream.spec.mjs` | Imports the verdict spec (`package.json`, which lists unit files, is part of the WebUI bundle's source digest); visible-only tests cover `inconclusive` | +| `scripts/webui/*_tests.py` (verifier, summarizer) | Rewritten for variable `paired_runs` and the policy | +| `.github/workflows/ci.yml`, `Makefile` | The new Python test added to the helper-test lists; no timeout changed | +| `docs/webui-integration-matrix.md` | Performance row describes the new verdict | + +Commits: `744c11f4` (the change), `a7cc03fa` (`/* global URL */` for lint, test script made executable, one blank line removed). + +## 3. Technical Decisions + +### 3.1 Median with a sign-test interval + +Per-pair noise on this runner is heavy-tailed (kurtosis above 10 across 636 runs) and skewed (-0.7). The sign-test interval needs no variance estimate and no random bootstrap; its only input is `k`, the largest rank with `P(Binomial(n, 1/2) <= k - 1) <= alpha`. At `alpha` 0.01 per tail, `k` is 2 at n = 12 (98% interval `[x(2), x(11)]`, since `P(<= 1) = 13/4096`) and 13 at n = 40; at n = 6 no order statistic qualifies, which is why `validatePolicy` rejects a `min_pairs` too small for its `tail_alpha`. Because both languages sort the same doubles and the binomial sums are exact in a double up to n = 50 (hence `max_pairs <= 50` in validation), JavaScript and Python agree bit for bit, so the verifier can compare medians and bounds with `!=` instead of a tolerance. Rejected: mean with a t-interval (hurt by the heavy tails) and Hodges-Lehmann (assumes a symmetry the data lacks); both had similar power. + +### 3.2 Three outcomes, and what the 4% ceiling means + +- `investigate`: the interval lies wholly above the 2% budget. An interval touching 2.0 is not above it (fixture case `[2.0, 2.0]` is `within-target`). +- `within-target`: not `investigate`, and the upper bound is at most 4%. +- `continue` below the cap, `inconclusive` at 40 pairs. + +The 2% budget is unchanged. The 4% ceiling is a precision requirement, not a second budget: a pass must have been precise enough to see a slowdown well past 2%. Under the resampling model, 4% gives 0.15% null `inconclusive` per mode while 3% gives 2.4%, which breaks the 1% false-failure target. Passing only when the interval lies wholly below 2% was rejected because it needs far more pairs, a tradeoff the issue had already recorded. + +The cost of this asymmetry is in the PR's own table: a true 2% slowdown passes 81.6% of the time and a true 4% slowdown passes 12.9% of the time (78.6% `investigate`). The gate is built to catch effects of 5% and above (96.0%, 99.67% at 6%); between 2% and 5% it is a partial detector. The issue asked for at least 95% detection of the injected slowdown, which is why the acceptance injection was sized at about 6%. + +### 3.3 Resampling estimate + +Noise model: all 636 complete diagnostics artifacts on this runner (2026-09-15 to 2026-10-22, 6,360 pairs) centred on a true 0% change, iid pairs drawn from that pool plus a per-run shift N(0, 0.83^2). 0.83 is the between-run SD estimated from runs after #2119; the one-visible and two-visible means in one run are uncorrelated (r = -0.07). 20,000 simulated runs per row, sequential looks included: + +| True degradation | investigate | inconclusive | within-target | mean pairs | +|---|---|---|---|---| +| 0% | 0.06% | 0.15% | 99.79% | 13.5 | +| 2% | 11.4% | 7.0% | 81.6% | 20.4 | +| 4% | 78.6% | 8.5% | 12.9% | 21.6 | +| 5% | 96.0% | 2.3% | 1.6% | 17.1 | +| 6% | 99.67% | 0.27% | 0.07% | 14.1 | +| 8% | 100% | 0% | 0% | 12.4 | + +False failure on a true 0% change is 0.21% per mode, about 0.4% per CI job (two modes) and about 0.6% in `full` mode; with iid pairs and no run shift it is below 0.01%. The run shift dominates: at a shift SD of 1.5 false `investigate` rises to 2.4%. + +### 3.4 Sequential looks, replayed by the validators + +Looks happen only at `min_pairs + k * pair_step` (12, 14, ..., 40); `lookVerdict` returns `continue` at any other count. The harness stops at the first decision. `sequentialVerdict` replays every look over a recorded sequence and throws if the samples continued past a decision ("continued after the verdict") or stopped without one ("without a verdict"), so a recorded run that kept sampling after its decision, or stopped before one, is rejected. The resampling estimate includes the sequential looks, so the 0.21% already pays for the repeated testing. + +### 3.5 One policy, three enforcers + +The harness, the verifier and the summarizer used to carry their own thresholds. Now the policy file is the single copy, validated identically in both languages (exact key set, `ceiling >= budget`, `0 < alpha < 0.5`, even `min_pairs` and `pair_step`, `(max - min) % step == 0`). The harness writes the policy into its output, and both Python validators reject evidence produced under a different policy. The verifier, which has the samples, re-derives each mode with the Python port; the summarizer, which only sees summaries, checks that the status follows from the recorded interval, that the median lies inside it, and that `inconclusive` occurs only at the cap. Sample and preflight counts come from `expected_counts(paired_runs)`: one warmup, one `off` and one mode sample per pair, and preflight records equal to the mode's client count (1 for one-visible and hidden, 2 for two-visible) times `1 + paired_runs`. + +### 3.6 Reporting and completion status + +Every failing mode is reported as `mode: status (slowdown | measurement noise): median, 98% interval, pair count, rule`, both on the harness's stdout and in the verifier's assertion, so the job log separates `investigate` from `inconclusive` without downloading the artifact. In `full` mode `performanceCompletion` ranks `investigate` above `inconclusive` above `within-target`; under `visible-only-headed`, which CI runs, the top-level status stays `incomplete` and the per-mode verdict is what fails the job. + +### 3.7 Warmup and wall time + +The data show no first-pair bias (pair 0 mean -0.50 against -0.48 overall, no order effect), so no leading pairs are discarded. The existing warmup request runs 1.6% slower than the first baseline and was already excluded. Per pair, one-visible takes 7.2 s median (7.7 s max) and two-visible 8.4 s (9.2 s max), plus about 34 s before the first pair. In CI the worst case at 40 pairs per mode is about 12 minutes and the expected null run about 4 minutes (about 2 before); `full` mode's worst case is about 20 minutes, inside the 1800 s activity timeout. The issue set a 15-minute worst-case target. + +## 4. Verification + +Local checks from the PR test plan: `pnpm --dir webui run unit` and `lint`, `make verify-webui-helper-tests` (including `activity_performance_verdict_tests.py`), the verifier and summarizer test scripts, and `make verify-versions verify-env-var-docs verify-python-tooling`. + +Fixture: the per-sample logs of every failing attempt named in #1925 and its comments (35423171315 a1; 35425297407 a1, a2, a3; 35425050365 a1; 35421166203 a1; 35567777700 a1 for #1944; 35588958676 a1 for #1946; 38265665905 a1, a2 for #2279) were copied before the artifacts expired. Concatenated in run order, their pairs decide `within-target` at 12 pairs in both modes (one-visible median -1.31%, interval [-4.50%, 1.83%]; two-visible -0.15%, [-4.99%, 2.45%]). With the Activity arm 6% slower the same pairs decide `investigate` at 14 pairs, and a drift case with baseline CV 12.19% decides `within-target`. + +On `lablup-dgxspark21`: + +| Run | Result | +|---|---| +| 39853154267 (test plan) | one-visible `within-target` (median -0.15%, [-1.83%, 1.25%]), two-visible `within-target` (-0.76%, [-2.64%, 0.47%]), 12 pairs each; verifier and summarizer accepted the evidence; verifier took 3 min 42 s | +| 39853154267, attempts 1 to 5 (jobs 120084449655, 120097378392, 120110916929, 120116442394, 120121248222) | Five consecutive green `WebUI installed artifact` runs at head `a7cc03fa` | +| 39870659579 on draft PR #2554 (closed, never merged) | Injected slowdown: one-visible `investigate` (median 4.58%, [3.89%, 5.68%]), two-visible `investigate` (8.05%, [7.11%, 9.62%]), 12 pairs each | + +The injection sleeps 400 us per decode tick for each open `/ui-api/v1/events` stream, which the Activity page holds open exactly while visible. + +Injections that did not work on GB10: + +- A per-poll CPU burst of 80 spinning threads (run 39862888632) raised runnable tasks to 46 to 87 in the visible arms, but decode stayed at 152 tok/s in both arms. +- A delay keyed on Activity polls (runs 39866915190 and 39868894929) measured 0%. The instrumented run showed that no operations poll landed inside a decode window, because each poll was answered the moment a request finished. + +## 5. Unverified and Open Items + +- **True effect of the injection not pre-measured.** The issue's criterion asked for an injection "whose true effect is measured beforehand at 4 to 8 percent". A local build on the shared CI host was not permitted, so the effect is characterised only by the gate's own output: one-visible (4.58%, [3.89%, 5.68%]) is inside 4 to 8%; two-visible's median (8.05%) and upper bound sit just above it. +- **The recorded fixture cases cannot return `investigate` by construction.** At their recorded five pairs, n is below `min_pairs` and the sign-test rank at n = 5 is 0, so every one returns `continue` with a null interval. The acceptance item "none of the six runs returns `investigate`" is therefore satisfied structurally; the substantive null evidence is the concatenated 12-pair sequences and the resampling table. +- **Guarantee depends on run-level shifts.** The 0.21% false-failure figure holds while between-run shifts stay near the measured 0.83 points; at 1.5 it is 2.4%. Keeping host contention out is the job of the #1949 host-quiet precheck, not of this verdict. +- **2% to 5% slowdowns are often passed** (81.6% at a true 2%, 12.9% at 4%), as described in 3.2. +- **`full` mode not exercised.** The native hidden acceptance is still user-deferred; its 20-minute worst case is an estimate from per-pair timings, not a measured run. +- **CI wall time roughly doubles on a null run** (about 4 minutes expected, about 2 before); the measured acceptance run took 3 min 42 s in the verifier. `ci.yml` timeouts are unchanged. +- **Rule text formatting.** Python's `f"{value:.2f}"` stands in for JavaScript's `toFixed(2)` "for the magnitudes seen here"; the verifier does not compare the `rule` string, so a divergence would be cosmetic only. + +## 6. Learning Points + +- A fixed-count gate against a small budget fails on noise by design; size the sample from the measured noise and let the interval decide when to stop, and make "could not decide" a distinct, named failure so nobody reads noise as a regression. +- Pick a statistic both implementations can compute identically. Order statistics with an exact binomial rank need no randomness and no tolerance, which lets a Python verifier re-derive a JavaScript verdict with exact equality. +- When a rule is enforced in several places, move its constants into one file, have the producer stamp the policy into its output, and have consumers reject evidence produced under another policy. +- An injected slowdown has to share a resource with the thing being measured at the moment it is measured. On GB10, CPU pressure did not slow decode, and Activity polls were answered between requests, never during a decode window, so a poll-keyed delay measured 0%. A delay tied to the event stream the page keeps open during decode did reach it. +- Copy evidence that a fixture depends on into the repository before CI artifacts expire. diff --git a/TECHNICAL_REPORTS/2553-activity-gate-interval-verdict-20261023.ko.md b/TECHNICAL_REPORTS/2553-activity-gate-interval-verdict-20261023.ko.md new file mode 100644 index 000000000..ef9af9c6a --- /dev/null +++ b/TECHNICAL_REPORTS/2553-activity-gate-interval-verdict-20261023.ko.md @@ -0,0 +1,124 @@ +# PR #2553: Activity decode 오버헤드를 5쌍이 아니라 구간으로 판정 + +**날짜**: 2026-10-23 +**상태**: PR 열림; Closes #1925 + +## 요약 + +`WebUI installed artifact` 잡의 Activity 성능 게이트는 모드마다 off/on을 번갈아 5쌍 측정하고, 쌍별 성능 저하의 중앙값이 2% 예산을 넘거나 baseline CV가 5%를 넘으면 그 모드를 실패시켰다. 자체 호스팅 GB10 러너 `lablup-dgxspark21`에서는 쌍별 노이즈(SD 3.6 포인트, kurtosis 12~25) 때문에 코드가 바뀌지 않아도 두 조건이 모두 발동한다. #1925와 그 코멘트에는 decode에 영향을 줄 수 없는 PR에서 실패한 시도 10건이 기록되어 있고, 그중 하나는 Activity arm이 6.76% 더 빨랐는데도 CV 때문에 실패했다. PR #2553은 이 규칙을 순차적이고 분포 가정이 없는 검정으로 바꾼다. 점추정은 쌍별 저하의 중앙값, 구간은 98% sign-test 구간 `[x(k), x(n+1-k)]`이다. 각 모드는 12쌍에서 시작해 2쌍씩 최대 40쌍까지 늘리며, 구간이 `investigate`(구간 전체가 2% 위), `within-target`(상한이 4% 이하) 중 하나로 결정될 때까지 측정한다. 상한에 도달해도 결정되지 않으면 `inconclusive`가 되고, 측정 노이즈를 명시하는 메시지와 함께 실패한다. 모든 상수는 JavaScript 하네스와 Python 포트가 함께 읽는 정책 파일 하나로 옮겼다. verifier는 원시 샘플에서 모든 모드의 판정을 다시 도출하고, 두 언어 구현은 기록된 샘플 벡터로 만든 공유 fixture 하나로 테스트된다. 기록된 636개 실행을 리샘플링한 추정에서 실제 변화가 0%일 때 오탐 실패율은 모드당 0.21%, 실제 저하가 5% 이상일 때 검출률은 95% 이상이다. 러너에서는 변경되지 않은 트리로 다섯 번 연속 통과했고, 일회용 PR에 주입한 slowdown은 두 모드 모두 `investigate`로 실패했다. 이슈가 제안한 주입 방식(CPU burst, Activity poll에 묶은 지연)은 GB10에서 decode에 전혀 닿지 않았다. 15개 파일, +3995/-120이며 그중 3278줄이 fixture다. + +## 1. 문제 정의 + +게이트의 판정은 `webui/scripts/activity-performance.mjs`에 있었다. 모드당 5쌍, `status: degradation > 2 || cv > 5 ? 'investigate' : 'within-target'`. 같은 규칙이 임계값과 개수를 각자 복사한 채 두 군데에서 더 적용되었다. `scripts/webui/verify_activity_performance.py`는 `paired_runs == 5`를 요구하고 `degradation > 2 or cv > 5`를 다시 적용했으며 모드별 샘플 수와 preflight 수를 하드코딩했다. `scripts/webui/summarize_activity_evidence.py`는 status, `paired_runs == 5`, `<= 2`, `<= 5` 검사를 반복했다. + +이 규칙에서 두 가지 결함이 나왔다. + +- **5쌍으로는 2%를 구분할 수 없다.** 실패한 실행들의 쌍별 범위는 6~21 포인트였다. 변화가 없는 변경도 중앙값 임계를 넘었다. PR #1920(표시 이름 문자열만 변경)은 one-visible +3.38%로, PR #2279(YAML 파서만 변경)는 두 번째 시도에서 one-visible +4.65%로 실패했다. +- **CV는 오버헤드가 아니라 drift를 측정했다.** baseline CV는 대부분 블록 전체에 걸친 느린 drift를 기록하는데, 번갈아 측정하는 쌍 구조는 바로 이 drift를 상쇄하기 위한 것이다. run 35425050365 attempt 1에서는 two-visible 블록 동안 `off` 속도가 171.4에서 141.7 tok/s로 떨어졌고, Activity arm이 6.76% 더 빨랐는데도 CV 7.45 때문에 실패했다. PR #2279 attempt 1은 모든 쌍이 더 빨랐는데 CV 5.17로 실패했다. + +단일 GB10 러너에서 재실행 한 번에 10~40분이 든다. 이슈는 2%나 5% 임계값을 올리는 방안을 기각했다. 노이즈와 실제 회귀를 똑같이 가리기 때문이다. 이슈의 maintainer 코멘트는 호스트 수준 경합(같은 머신의 동시 컴파일로, GPU 프로세스 검사에는 보이지 않는다)도 점수를 움직였다고 덧붙였다(예: PR #1944, #1946). 그 부분은 #1949(host-quiet precheck)로 분리되었고, 이 PR은 판정만 다룬다. + +## 2. 변경 요약 + +| 영역 | 변경 | +|---|---| +| `webui/scripts/activity-performance-policy.json` (신규) | 상수의 유일한 사본: `budget_percent` 2, `precision_ceiling_percent` 4, `tail_alpha` 0.01, `min_pairs` 12, `pair_step` 2, `max_pairs` 40 | +| `webui/scripts/activity-performance-config.mjs` | 순수 판정 함수: `validatePolicy`, `verdictPolicy`(import 시 로드 및 검증), `signTestRank`, `pairDegradation`, `medianInterval`, `lookVerdict`, `sequentialVerdict`, `verdictRule`, `modeSummary`, `pairsFromSamples`; `performanceCompletion`에 `inconclusive` 추가 | +| `webui/scripts/activity-performance.mjs` | `lookVerdict`가 `continue`를 벗어날 때까지 측정 루프 반복; 모드마다 `mode: status: rule` 출력; 출력에 `verdict_policy` 기록; 기존 `median` 헬퍼(n이 짝수면 위쪽 가운데 원소를 취함) 제거 | +| `scripts/webui/activity_performance_verdict.py` (신규) | 같은 함수들을 연산 단위로 옮긴 Python 포트, 그리고 `check_summary_consistency`, `failure_message`, `expected_counts` | +| `scripts/webui/verify_activity_performance.py` | `samples`에서 모드별 판정을 재도출; `status`, `paired_runs`, 중앙값, 구간은 정확히 일치해야 하고 CV와 paired range는 상대 오차 1e-9 허용; 샘플 수와 preflight 수를 `paired_runs`에서 도출해 하드코딩된 두 개수 표를 대체 | +| `scripts/webui/summarize_activity_evidence.py` | 샘플이 없으므로 정책 기준의 구조와 내부 일관성만 검사; `verdict_policy`가 저장소의 정책과 같아야 함; `interval_percent`와 `verdict_policy`를 출력; CV 상한 제거 | +| `webui/scripts/activity-performance-fixtures.json` (신규) | 22개 케이스: 기록된 실패 시도 10건, 이어 붙인 null 시퀀스 2건, 합성 slowdown, speedup, drift, inconclusive, 경계 케이스, 거부 케이스 4건 | +| `webui/scripts/activity-performance-verdict.spec.mjs` (신규), `scripts/webui/activity_performance_verdict_tests.py` (신규) | 둘 다 모든 fixture 케이스를 평가해 `expected`를 정확히 재현해야 함; JS는 sign-test rank, look 일정, 정책 검증, `performanceCompletion`도 테스트 | +| `webui/scripts/activity-stream.spec.mjs` | verdict spec을 import(unit 파일 목록을 가진 `package.json`이 WebUI 번들 source digest에 포함되기 때문); visible-only 테스트가 `inconclusive`도 다룸 | +| `scripts/webui/*_tests.py` (verifier, summarizer) | 가변 `paired_runs`와 정책에 맞춰 재작성 | +| `.github/workflows/ci.yml`, `Makefile` | 새 Python 테스트를 helper 테스트 목록에 추가; 타임아웃 변경 없음 | +| `docs/webui-integration-matrix.md` | Performance 행에 새 판정 설명 | + +커밋: `744c11f4`(본 변경), `a7cc03fa`(lint용 `/* global URL */`, 테스트 스크립트 실행 권한, 빈 줄 하나 제거). + +## 3. 기술적 선택과 그 이유 + +### 3.1 중앙값과 sign-test 구간 + +이 러너의 쌍별 노이즈는 꼬리가 두껍고(636개 실행에서 kurtosis 10 초과) 치우쳐 있다(skew -0.7). sign-test 구간은 분산 추정도, 무작위 bootstrap도 필요 없다. 입력은 `P(Binomial(n, 1/2) <= k - 1) <= alpha`를 만족하는 가장 큰 rank `k` 하나뿐이다. 꼬리당 `alpha` 0.01에서 n = 12이면 `k`는 2이고(`P(<= 1) = 13/4096`이므로 98% 구간은 `[x(2), x(11)]`), n = 40이면 13이다. n = 6에서는 조건을 만족하는 순서 통계량이 없으므로 `validatePolicy`는 `tail_alpha`에 비해 너무 작은 `min_pairs`를 거부한다. 두 언어가 같은 double을 정렬하고 이항 합은 n = 50까지 double에서 정확하므로(그래서 검증에 `max_pairs <= 50`이 있다) JavaScript와 Python 결과가 비트 단위로 같다. 그 덕에 verifier는 중앙값과 구간 경계를 허용 오차 없이 `!=`로 비교한다. 기각한 대안: t-구간을 쓰는 평균(두꺼운 꼬리에 취약)과 Hodges-Lehmann(데이터에 없는 대칭성을 가정). 둘 다 검정력은 비슷했다. + +### 3.2 세 가지 결과와 4% 상한의 의미 + +- `investigate`: 구간 전체가 2% 예산보다 위에 있다. 2.0에 닿기만 하는 구간은 위에 있는 것이 아니다(fixture의 `[2.0, 2.0]` 케이스는 `within-target`). +- `within-target`: `investigate`가 아니고 상한이 4% 이하다. +- 상한 전에는 `continue`, 40쌍에서는 `inconclusive`. + +2% 예산은 그대로다. 4% 상한은 두 번째 예산이 아니라 정밀도 요건이다. 통과하려면 2%를 충분히 넘는 slowdown을 볼 수 있을 만큼 측정이 정밀했어야 한다. 리샘플링 모델에서 4%는 null `inconclusive`가 모드당 0.15%이고, 3%는 2.4%로 1% 오탐 목표를 깬다. 구간 전체가 2% 아래에 있을 때만 통과시키는 방안은 훨씬 많은 쌍이 필요해서 기각했고, 이 트레이드오프는 이슈에 이미 기록되어 있었다. + +이 비대칭의 대가는 PR 자체 표에 나온다. 실제 2% slowdown은 81.6% 확률로, 실제 4% slowdown은 12.9% 확률로 통과한다(78.6%는 `investigate`). 게이트는 5% 이상 효과(96.0%, 6%에서 99.67%)를 잡도록 설계되었고, 2%와 5% 사이에서는 부분적인 검출기다. 이슈가 주입 slowdown에 대해 95% 이상 검출을 요구했기 때문에 acceptance 주입을 약 6%로 잡았다. + +### 3.3 리샘플링 추정 + +노이즈 모델: 이 러너의 완전한 diagnostics artifact 636개 전체(2026-09-15~2026-10-22, 6,360쌍)를 실제 변화 0%로 중심화하고, 그 풀에서 iid로 쌍을 뽑은 뒤 실행별 shift N(0, 0.83^2)를 더했다. 0.83은 #2119 이후 실행에서 추정한 실행 간 SD이고, 한 실행 안의 one-visible 평균과 two-visible 평균은 상관이 없다(r = -0.07). 행마다 20,000회 시뮬레이션, 순차 look 포함: + +| 실제 저하 | investigate | inconclusive | within-target | 평균 쌍 수 | +|---|---|---|---|---| +| 0% | 0.06% | 0.15% | 99.79% | 13.5 | +| 2% | 11.4% | 7.0% | 81.6% | 20.4 | +| 4% | 78.6% | 8.5% | 12.9% | 21.6 | +| 5% | 96.0% | 2.3% | 1.6% | 17.1 | +| 6% | 99.67% | 0.27% | 0.07% | 14.1 | +| 8% | 100% | 0% | 0% | 12.4 | + +실제 변화 0%에서 오탐 실패는 모드당 0.21%, CI 잡당(두 모드) 약 0.4%, `full` 모드에서 약 0.6%다. 쌍이 iid이고 실행 shift가 없으면 0.01% 미만이다. 지배적인 요인은 실행 shift다. shift SD가 1.5이면 오탐 `investigate`가 2.4%로 오른다. + +### 3.4 순차 look, 그리고 validator의 재생 + +look은 `min_pairs + k * pair_step`(12, 14, ..., 40)에서만 일어나고, 그 외의 개수에서 `lookVerdict`는 `continue`를 반환한다. 하네스는 첫 결정에서 멈춘다. `sequentialVerdict`는 기록된 시퀀스에 대해 모든 look을 다시 재생하고, 결정 이후에도 계속 측정했거나("continued after the verdict") 결정 없이 멈춘("without a verdict") 기록을 거부한다. 리샘플링 추정이 순차 look을 포함하므로 0.21%에는 반복 검정 비용이 이미 반영되어 있다. + +### 3.5 정책 하나, 집행자 셋 + +하네스, verifier, summarizer는 원래 각자 임계값을 갖고 있었다. 이제 정책 파일이 유일한 사본이고 두 언어에서 같은 방식으로 검증된다(정확한 키 집합, `ceiling >= budget`, `0 < alpha < 0.5`, 짝수 `min_pairs`와 `pair_step`, `(max - min) % step == 0`). 하네스는 출력에 정책을 기록하고, 두 Python validator는 다른 정책으로 만든 증거를 거부한다. 샘플을 가진 verifier는 Python 포트로 모드마다 판정을 재도출한다. 요약만 보는 summarizer는 status가 기록된 구간에서 따라 나오는지, 중앙값이 구간 안에 있는지, `inconclusive`가 상한에서만 나오는지를 검사한다. 샘플 수와 preflight 수는 `expected_counts(paired_runs)`에서 나온다. warmup 1개, 쌍마다 `off` 샘플 하나와 모드 샘플 하나, preflight 기록은 모드의 클라이언트 수(one-visible과 hidden은 1, two-visible은 2)에 `1 + paired_runs`를 곱한 값이다. + +### 3.6 보고 방식과 완료 상태 + +실패한 모드는 `mode: status (slowdown | measurement noise): 중앙값, 98% 구간, 쌍 수, 규칙` 형식으로 하네스 stdout과 verifier assertion 양쪽에 보고된다. 그래서 artifact를 내려받지 않고도 잡 로그에서 `investigate`와 `inconclusive`를 구별할 수 있다. `full` 모드에서 `performanceCompletion`은 `investigate`, `inconclusive`, `within-target` 순으로 우선한다. CI가 돌리는 `visible-only-headed`에서는 최상위 status가 `incomplete`로 유지되고, 잡을 실패시키는 것은 모드별 판정이다. + +### 3.7 Warmup과 소요 시간 + +데이터에는 첫 쌍 편향이 없으므로(pair 0 평균 -0.50 대 전체 -0.48, 순서 효과 없음) 앞쪽 쌍을 버리지 않는다. 기존 warmup 요청은 첫 baseline보다 1.6% 느리며 이미 제외되어 있었다. 쌍당 one-visible은 중앙값 7.2초(최대 7.7초), two-visible은 8.4초(최대 9.2초)가 걸리고, 첫 쌍 전에 약 34초가 더해진다. CI에서 모드당 40쌍인 최악의 경우는 약 12분, 예상 null 실행은 약 4분(이전에는 약 2분)이다. `full` 모드의 최악은 약 20분으로 1800초 activity 타임아웃 안이다. 이슈의 최악 목표는 15분 미만이었다. + +## 4. 검증 + +PR test plan의 로컬 검사: `pnpm --dir webui run unit`과 `lint`, `make verify-webui-helper-tests`(`activity_performance_verdict_tests.py` 포함), verifier와 summarizer 테스트 스크립트, `make verify-versions verify-env-var-docs verify-python-tooling`. + +Fixture: #1925와 그 코멘트에 나온 모든 실패 시도(35423171315 a1; 35425297407 a1, a2, a3; 35425050365 a1; 35421166203 a1; #1944의 35567777700 a1; #1946의 35588958676 a1; #2279의 38265665905 a1, a2)의 샘플별 로그를 artifact가 만료되기 전에 복사했다. 실행 순서대로 이어 붙이면 두 모드 모두 12쌍에서 `within-target`으로 결정된다(one-visible 중앙값 -1.31%, 구간 [-4.50%, 1.83%]; two-visible -0.15%, [-4.99%, 2.45%]). 같은 쌍에서 Activity arm을 6% 느리게 하면 14쌍에서 `investigate`, baseline CV 12.19%인 drift 케이스는 `within-target`으로 결정된다. + +`lablup-dgxspark21`에서: + +| 실행 | 결과 | +|---|---| +| 39853154267 (test plan) | one-visible `within-target`(중앙값 -0.15%, [-1.83%, 1.25%]), two-visible `within-target`(-0.76%, [-2.64%, 0.47%]), 각 12쌍; verifier와 summarizer가 증거 수용; verifier 소요 3분 42초 | +| 39853154267, attempt 1~5 (job 120084449655, 120097378392, 120110916929, 120116442394, 120121248222) | head `a7cc03fa`에서 `WebUI installed artifact` 다섯 번 연속 통과 | +| draft PR #2554의 39870659579 (닫힘, 머지되지 않음) | 주입 slowdown: one-visible `investigate`(중앙값 4.58%, [3.89%, 5.68%]), two-visible `investigate`(8.05%, [7.11%, 9.62%]), 각 12쌍 | + +주입은 열린 `/ui-api/v1/events` 스트림마다 decode tick당 400 us를 sleep한다. Activity 페이지는 보이는 동안에만 정확히 이 스트림을 열어 둔다. + +GB10에서 작동하지 않은 주입: + +- poll마다 80개 스레드를 돌리는 CPU burst(run 39862888632)는 visible arm의 runnable task를 46~87로 올렸지만 decode는 두 arm 모두 152 tok/s에 머물렀다. +- Activity poll에 묶은 지연(run 39866915190, 39868894929)은 0%로 측정되었다. 계측한 실행을 보면 operations poll이 decode 구간 안에 들어온 적이 한 번도 없었다. 각 poll은 요청이 끝나는 순간에 응답되었다. + +## 5. 미검증 및 미해결 항목 + +- **주입의 실제 효과를 미리 측정하지 않았다.** 이슈의 기준은 "실제 효과를 미리 4~8%로 측정한" 주입이었다. 공유 CI 호스트에서 로컬 빌드가 허용되지 않아 효과는 게이트 자체 출력으로만 특성화되었다. one-visible(4.58%, [3.89%, 5.68%])은 4~8% 안이고, two-visible의 중앙값(8.05%)과 상한은 그보다 조금 위다. +- **기록된 fixture 케이스는 구조상 `investigate`를 반환할 수 없다.** 기록된 5쌍에서는 n이 `min_pairs`보다 작고 n = 5의 sign-test rank가 0이라, 모두 구간이 null인 `continue`를 반환한다. 따라서 "여섯 실행 중 어느 것도 `investigate`를 반환하지 않는다"는 acceptance 항목은 구조적으로 충족된다. 실질적인 null 증거는 이어 붙인 12쌍 시퀀스와 리샘플링 표다. +- **보장은 실행 수준 shift에 의존한다.** 0.21% 오탐 수치는 실행 간 shift가 측정값 0.83 포인트 근처에 머무는 동안만 성립하고, 1.5에서는 2.4%다. 호스트 경합을 막는 것은 이 판정이 아니라 #1949 host-quiet precheck의 몫이다. +- **2~5% slowdown은 자주 통과한다**(실제 2%에서 81.6%, 4%에서 12.9%). 3.2 참고. +- **`full` 모드는 실행되지 않았다.** native hidden acceptance는 여전히 사용자 보류 상태이고, 20분 최악 시간은 쌍당 시간에서 낸 추정이지 측정값이 아니다. +- **null 실행의 CI 소요 시간이 대략 두 배가 된다**(예상 약 4분, 이전 약 2분). 측정된 acceptance 실행에서 verifier는 3분 42초 걸렸다. `ci.yml` 타임아웃은 그대로다. +- **규칙 문자열 포맷.** Python의 `f"{value:.2f}"`가 JavaScript의 `toFixed(2)`를 "여기서 보이는 크기 범위에서" 대신한다. verifier는 `rule` 문자열을 비교하지 않으므로 차이가 나도 표시상의 문제에 그친다. + +## 6. 학습 포인트 + +- 작은 예산에 대해 고정 개수로 판정하는 게이트는 설계상 노이즈로 실패한다. 측정된 노이즈에서 표본 크기를 정하고, 언제 멈출지는 구간이 결정하게 하라. "결정할 수 없음"은 이름을 가진 별도의 실패로 두어 누구도 노이즈를 회귀로 읽지 않게 한다. +- 두 구현이 똑같이 계산할 수 있는 통계량을 고르라. 정확한 이항 rank를 쓰는 순서 통계량은 무작위성도 허용 오차도 필요 없어서, Python verifier가 JavaScript 판정을 정확한 일치로 재도출할 수 있다. +- 한 규칙이 여러 곳에서 집행되면 상수를 파일 하나로 옮기고, 생산자가 출력에 정책을 찍게 하고, 소비자는 다른 정책으로 만든 증거를 거부하게 하라. +- 주입한 slowdown은 측정 대상과 측정 순간에 같은 자원을 공유해야 한다. GB10에서 CPU 압박은 decode를 늦추지 않았고, Activity poll은 decode 구간이 아니라 요청 사이에 응답되어 poll에 묶은 지연은 0%였다. 페이지가 decode 동안 열어 두는 이벤트 스트림에 묶은 지연은 decode에 닿았다. +- fixture가 의존하는 증거는 CI artifact가 만료되기 전에 저장소로 복사하라. diff --git a/docs/webui-integration-matrix.md b/docs/webui-integration-matrix.md index 7f8dc7346..f43744108 100644 --- a/docs/webui-integration-matrix.md +++ b/docs/webui-integration-matrix.md @@ -14,7 +14,7 @@ This matrix is the canonical handoff for the bundled WebUI release gate. Mock/un | State completion and races | Existing contract/unit suites plus `verify-webui-rust` cover bad transitions, operation list, restart/gap fixtures, download cancellation and schema errors; the real-router harness additionally exercises empty-library rescan, real same-port fresh server restart SSE, event-ring gap SSE, stale revision 409, unknown-field 400 and unsupported terminal-operation cancel on the production secured route. | PASS for the real-router fake harness at `ff4463c6`; unit/aggregate rerun remains root-owned. | Preserve failed attempts and exact commit/binary identity in the final evidence table. | | Browser visual/accessibility engines | `pnpm --dir webui run browser` keeps the approved Chromium pixel-baseline/Chromium-AX suite, while `pnpm --dir webui run browser:all` uses `playwright.engines.config.ts` for Chromium/Firefox/WebKit DOM, keyboard, layout and axe coverage without invoking Chromium-only CDP or screenshot baselines. CI installs all three engines and runs both paths through `verify-webui-frontend`. | Root PASS on mock/Vite Chromium evidence: 28 default Chromium tests, 7 cross-surface/performance Chromium tests in `integration-1848/engines-f873-quiet-pass`, `chromium-prefix-default-pass`, and latest true DOM+paint timer evidence in `integration-1848/engines-browser-only-timing-pass`; 311 Vitest, 17 Node, 49 contract tests, typecheck and lint also passed. | **Native Safari on the installed artifact: USER-REPORTED PASS 2026-09-16**, against `mlxcel-server --webui` serving the embedded bundle over a loopback-forwarded session, not a Vite preview. Scope confirmed by the maintainer, and only this scope: screen and layout reviewed by eye across Models, Chat, Activity and Settings; keyboard Tab and Shift-Tab order with visible focus and Escape closing dialogs and sheets with focus restored to the opener; Safari native page zoom at 200% with no horizontal page scroll at 390 px, and readability retained with increased contrast. **Manual VoiceOver: user-deferred 2026-09-16, NOT RUN.** It was not part of the Safari pass and no other gate substitutes for it, since the axe and DOM checks in CI assert markup rather than what a screen reader announces. Deferred by the maintainer on the stated grounds that the UI has more pressing changes outstanding, so this does not block #1848 or the epic. Scope of what remains unverified: whether VoiceOver announces the primary navigation and selected route, the command dialog title, the model combobox, status badges, streaming summaries and error text. Lifting the deferral needs a VoiceOver session against the installed artifact; no automated gate can close this row. All-three-engine CI is covered by the `WebUI bundle` job. This supersedes the 2026-09-13 report only for the surfaces above; that one covered the design-system candidate under a Vite preview at `#gallery`. | | Bundle budgets and deterministic assets | `make verify-webui-bundle` still enforces initial JS, total JS, embedded bundle size, deterministic checked-in assets, no source maps/local paths/active external origins, and license notice. | PASS on the integrated source (main `b6d69bcc` plus the #1900 docs/CLI branch, no asset change): CI `WebUI bundle` job (https://github.com/lablup/mlxcel/actions/runs/38785163375/job/116595293970) ran `build_bundle.py --verify` and printed `WebUI bundle verified: digest=b4087a9339fa572f260745cf5cd2af5ed9622b51353f6cab1532099072d332be`. The committed manifest records `source_digest_sha256` `1ebf0921089e00db6bf288cdd66fb2ba6265f9ee43a4224a07d483f1201d79c8`, 855,324 embedded bytes (limit 5,242,880), initial JS gzip 181,473 bytes (limit 204,800) and total JS gzip 182,086 bytes (limit 716,800). The CUDA `test-fast` `mlxcel-server` (SHA-256 `575f67ed7a87e7afb3bcdd431247d64d2e93e2295b2777816700a7d8675efcaa`) and `mlxcel` (`3ca9cdd875d5bb676346c8b6627bb531deb0bf2c4db014d4b4e15a9fe5cd6a17`) built for #1900 embed these hashed assets. | A release rebuild must rerun `make verify-webui-bundle` and cite its own job if any WebUI source or asset changes after this row. The initial JS gzip limit has since moved from 204,800 to 262,144 bytes (256 KiB) for the ui-common 0.2 foundations; the figures in this row predate that change. | -| Performance and large data | Existing frontend tests cover 1,000-entry catalog and 10,000-token transcript parsing; `make verify-webui-startup` records installed empty-router startup/RSS; `make verify-webui-activity-performance` runs the real Activity overhead/native-hidden harness with a checkpoint, including the genuine Linux GB10 headed Chrome minimize-to-`document.hidden` path when invoked with the approved host/display setup. | Root PASS evidence: installed startup/RSS `integration-1848/startup-52103-pass`; mock/Vite Chromium true DOM+paint timing `integration-1848/engines-browser-only-timing-pass` recorded cold 59.8 ms, navigation 25.8 ms, 1,000-entry search 25.9 ms, 10,000-token new conversation 32.0 ms, CLS 0. Prior visible Activity evidence is partial, tied to its source commit only. The GB10 job now runs the headed visible-mode observation overhead on every change (`--perf-mode visible-only-headed`) and summarizes it with `--expect-hidden-native not-run`, so the summary declares `status: incomplete` and `hidden_native: not-run` rather than reading as a complete acceptance. | **Native hidden acceptance: user-deferred 2026-09-16.** Scope of the deferral is the minimize-to-`document.hidden` measurement only; the headed visible-mode overhead is measured in CI and every numeric budget still applies to it. Reason, established on the GB10 runner and not inferred: Xvfb and openbox give headed Chromium a real 700x900 content area, the browser reports the window state as `minimized`, and `document.visibilityState` stays `visible` for 15 seconds anyway; foregrounding a sibling tab in the same context also leaves `document.hidden` false; both visibility-suppressing launch arguments (`--disable-backgrounding-occluded-windows`, `--disable-renderer-backgrounding`) were removed and their removal verified by reading a live browser process's arguments. The unverified step in that chain is whether Playwright's sibling page is a tab in the same window or a separate window, which would make the tab datum uninformative. This is a property of the automation browser, not of the runner or the product. The visibility backoff the row exists to protect stays covered by two unit tests in `webui/src/state/sync.test.ts` and the `visibilitychange` case in `webui/tests/browser.spec.ts`. Lifting the deferral means running `--perf-mode full` again, which needs a browser that propagates a real window-visibility change. | +| Performance and large data | Existing frontend tests cover 1,000-entry catalog and 10,000-token transcript parsing; `make verify-webui-startup` records installed empty-router startup/RSS; `make verify-webui-activity-performance` runs the real Activity overhead/native-hidden harness with a checkpoint, including the genuine Linux GB10 headed Chrome minimize-to-`document.hidden` path when invoked with the approved host/display setup. Each Activity mode's verdict (#1925) judges the paired decode degradation `(off - on) / off` by its median and a distribution-free 98% sign-test interval: `investigate` (a slowdown) when the interval lies wholly above the 2% budget, `within-target` when it does not and its upper bound is at most 4%, otherwise more alternating pairs, from 12 in steps of 2 up to 40, and then `inconclusive` (measurement noise, not a slowdown). The constants live in `webui/scripts/activity-performance-policy.json`, shared by the harness and both Python validators, which re-derive or cross-check the verdict against `webui/scripts/activity-performance-fixtures.json`; baseline CV and the paired range are diagnostics only. | Root PASS evidence: installed startup/RSS `integration-1848/startup-52103-pass`; mock/Vite Chromium true DOM+paint timing `integration-1848/engines-browser-only-timing-pass` recorded cold 59.8 ms, navigation 25.8 ms, 1,000-entry search 25.9 ms, 10,000-token new conversation 32.0 ms, CLS 0. Prior visible Activity evidence is partial, tied to its source commit only. The GB10 job now runs the headed visible-mode observation overhead on every change (`--perf-mode visible-only-headed`) and summarizes it with `--expect-hidden-native not-run`, so the summary declares `status: incomplete` and `hidden_native: not-run` rather than reading as a complete acceptance. | **Native hidden acceptance: user-deferred 2026-09-16.** Scope of the deferral is the minimize-to-`document.hidden` measurement only; the headed visible-mode overhead is measured in CI and the decode-overhead verdict still applies to it. Reason, established on the GB10 runner and not inferred: Xvfb and openbox give headed Chromium a real 700x900 content area, the browser reports the window state as `minimized`, and `document.visibilityState` stays `visible` for 15 seconds anyway; foregrounding a sibling tab in the same context also leaves `document.hidden` false; both visibility-suppressing launch arguments (`--disable-backgrounding-occluded-windows`, `--disable-renderer-backgrounding`) were removed and their removal verified by reading a live browser process's arguments. The unverified step in that chain is whether Playwright's sibling page is a tab in the same window or a separate window, which would make the tab datum uninformative. This is a property of the automation browser, not of the runner or the product. The visibility backoff the row exists to protect stays covered by two unit tests in `webui/src/state/sync.test.ts` and the `visibilitychange` case in `webui/tests/browser.spec.ts`. Lifting the deferral means running `--perf-mode full` again, which needs a browser that propagates a real window-visibility change. | | Actual inference/media | No committed mock gate can satisfy actual inference. Accepted Chat source9385 dense/hybrid/VLM evidence may be referenced only for the specific dense/hybrid/VLM generation paths whose source stayed identical through baseline6976 (`git diff --name-only 9385bcb7 6976e1df -- src Cargo.toml Cargo.lock build.rs webui/src webui/package.json webui/pnpm-lock.yaml` was empty). This is source-equivalence evidence for those paths, not an identical-binary or blanket current-branch runtime/UI claim: #1848 also includes production URL-prefix/public-health changes (`d6b7`, `c629`) plus test harness changes. | Root single-model installed four-arm Hello capture PASS in `integration-1848/single-model-52103-pass`, with semantic review scope limited to that small real checkpoint prompt and legacy own SIGINT `-2` noted; source-equivalent Chat dense/hybrid/VLM rows remain accepted only with the limitation above. | Root must still run or reference exact rows for small public download, VLM image, and genuine native-hidden Activity overhead with `MLXCEL_REQUIRE_MODELS=1`, recording checkpoint revisions, outputs, lifecycle order, and resource before/after. | | CUDA host smoke | `make verify-webui-cuda WEBUI_CUDA_EVIDENCE=...` validates the root-run CUDA evidence JSON for UI-on, UI-off and installed-artifact rows. | `not-run`; not satisfied by macOS. | GB10/CUDA host must run UI-on/UI-off server smoke and record result; historic outage waivers are not blanket permission. | | Settings / Chat / Activity cross-surface integration | Existing merged child gates provide source-specific evidence; this matrix points to final rerun rather than inheriting a fictional one-run pass. | Prior root notes: Models source22f19 lifecycle/profile/navigation/removal passed with generationTested=false; Chat source9385 dense/hybrid/VLM semantics, Stop active→0, CSP, unload passed; Activity visible paired overhead partial passed while native hidden did not run. | Final integrated artifact must rerun or explicitly reference accepted prior evidence by commit/source and limitation. Chat composer caps new turns at 100 while history import/storage permits 200 per conversation; Stop refreshes aggregate Activity, not a per-request cancellation receipt. | diff --git a/scripts/webui/activity_performance_verdict.py b/scripts/webui/activity_performance_verdict.py new file mode 100644 index 000000000..9c8ef607d --- /dev/null +++ b/scripts/webui/activity_performance_verdict.py @@ -0,0 +1,211 @@ +# Copyright 2026 Lablup Inc. Licensed under the Apache License, Version 2.0. +"""Python port of the Activity decode-overhead verdict (#1925). + +The rule and its rationale live in webui/scripts/activity-performance-config.mjs; every constant +lives in webui/scripts/activity-performance-policy.json, which this module reads instead of +keeping its own copy. Both implementations are tested against +webui/scripts/activity-performance-fixtures.json. Arithmetic follows the JavaScript operation +for operation, so the two produce bit-identical medians and interval bounds. +""" +from __future__ import annotations + +import json +import math +from pathlib import Path +from typing import Any + +POLICY_PATH = Path(__file__).resolve().parents[2] / "webui" / "scripts" / "activity-performance-policy.json" +FIXTURE_PATH = POLICY_PATH.with_name("activity-performance-fixtures.json") +POLICY_KEYS = ("budget_percent", "precision_ceiling_percent", "tail_alpha", "min_pairs", "pair_step", "max_pairs") +STATUSES = ("investigate", "within-target", "inconclusive") +# Browser clients per measured mode; with a variable pair count, sample and preflight counts are +# derived from each mode's recorded paired_runs by expected_counts(). +CLIENTS_PER_MODE = {"one-visible": 1, "two-visible": 2, "hidden": 1} + + + +def _is_int(value: Any) -> bool: + return isinstance(value, int) and not isinstance(value, bool) + + +def _is_number(value: Any) -> bool: + return isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) + + +def sign_test_rank(n: int, alpha: float) -> int: + """Largest k with P(Binomial(n, 1/2) <= k - 1) <= alpha; 0 when no order statistic qualifies.""" + cumulative = 0 + k = 0 + for j in range(n + 1): + cumulative += math.comb(n, j) + if cumulative > alpha * 2**n: + break + k = j + 1 + return k + + +def validate_policy(value: Any) -> dict[str, Any]: + if not isinstance(value, dict): + raise AssertionError("Activity verdict policy must be an object") + if sorted(value) != sorted(POLICY_KEYS): + raise AssertionError(f"Activity verdict policy must have exactly: {', '.join(POLICY_KEYS)}") + budget, ceiling, alpha = value["budget_percent"], value["precision_ceiling_percent"], value["tail_alpha"] + if not all(_is_number(v) for v in (budget, ceiling, alpha)) or not ceiling >= budget or not 0 < alpha < 0.5: + raise AssertionError("Activity verdict policy thresholds are invalid") + low, step, high = value["min_pairs"], value["pair_step"], value["max_pairs"] + if not all(_is_int(v) for v in (low, step, high)) or step < 2 or step % 2 or low % 2 or high > 50 or high < low or (high - low) % step: + raise AssertionError("Activity verdict policy pair counts are invalid") + if sign_test_rank(low, alpha) < 1: + raise AssertionError("Activity verdict policy min_pairs is too small for its tail_alpha") + return dict(value) + + +def load_policy(path: Path = POLICY_PATH) -> dict[str, Any]: + return validate_policy(json.loads(path.read_text(encoding="utf-8"))) + + +def pair_degradation(pair: dict[str, Any]) -> float: + off, on = pair.get("off"), pair.get("on") + if not (_is_number(off) and off > 0 and _is_number(on) and on > 0): + raise AssertionError("Activity pair rates must be finite and positive") + return (off - on) / off * 100 + + +def median_interval(values: list[float], policy: dict[str, Any]) -> tuple[float, list[float] | None]: + ordered = sorted(values) + n = len(ordered) + if n == 0: + raise AssertionError("Activity verdict needs at least one pair") + median = ordered[(n - 1) // 2] if n % 2 == 1 else (ordered[n // 2 - 1] + ordered[n // 2]) / 2 + k = sign_test_rank(n, policy["tail_alpha"]) + return median, (None if k < 1 else [ordered[k - 1], ordered[n - k]]) + + +def look_verdict(pairs: list[dict[str, Any]], policy: dict[str, Any]) -> dict[str, Any]: + n = len(pairs) + median, interval = median_interval([pair_degradation(p) for p in pairs], policy) + base = {"paired_runs": n, "median_decode_degradation_percent": median, "interval_percent": interval} + if n < policy["min_pairs"] or n > policy["max_pairs"] or (n - policy["min_pairs"]) % policy["pair_step"]: + return {**base, "status": "continue"} + assert interval is not None + if interval[0] > policy["budget_percent"]: + return {**base, "status": "investigate"} + if interval[1] <= policy["precision_ceiling_percent"]: + return {**base, "status": "within-target"} + return {**base, "status": "inconclusive" if n >= policy["max_pairs"] else "continue"} + + +def sequential_verdict(pairs: list[dict[str, Any]], policy: dict[str, Any]) -> dict[str, Any]: + n = policy["min_pairs"] + while n <= min(len(pairs), policy["max_pairs"]): + verdict = look_verdict(pairs[:n], policy) + if verdict["status"] != "continue": + if n != len(pairs): + raise AssertionError(f"Activity sampling continued after the verdict at {n} pairs") + return verdict + n += policy["pair_step"] + raise AssertionError(f"Activity sampling stopped at {len(pairs)} pairs without a verdict") + + +def _fixed(value: float) -> str: + # JavaScript's toFixed(2) for the magnitudes seen here; only used in human-readable text. + return f"{value:.2f}" + + +def verdict_rule(summary: dict[str, Any], policy: dict[str, Any]) -> str: + low, high = summary["interval_percent"] + confidence = round((1 - 2 * policy["tail_alpha"]) * 100) + head = f"median {_fixed(summary['median_decode_degradation_percent'])}%, {confidence}% interval [{_fixed(low)}%, {_fixed(high)}%] over {summary['paired_runs']} pairs" + if summary["status"] == "investigate": + return f"{head}; slowdown: the interval lies wholly above the {policy['budget_percent']}% budget" + if summary["status"] == "within-target": + return f"{head}; within target: the interval reaches no further than {policy['precision_ceiling_percent']}% and does not lie above the {policy['budget_percent']}% budget" + return f"{head}; measurement noise, not a slowdown: after the {policy['max_pairs']}-pair cap the interval still straddles the {policy['budget_percent']}% budget and reaches past {policy['precision_ceiling_percent']}%" + + +def mode_summary(mode: str, pairs: list[dict[str, Any]], policy: dict[str, Any]) -> dict[str, Any]: + verdict = sequential_verdict(pairs, policy) + degradations = [pair_degradation(p) for p in pairs] + baseline = [p["off"] for p in pairs] + total = 0.0 + for value in baseline: + total += value + mean = total / len(baseline) + squares = 0.0 + for value in baseline: + squares += (value - mean) ** 2 + cv = math.sqrt(squares / len(baseline)) / mean * 100 + summary = { + "mode": mode, + "status": verdict["status"], + "paired_runs": verdict["paired_runs"], + "median_decode_degradation_percent": verdict["median_decode_degradation_percent"], + "interval_percent": verdict["interval_percent"], + "baseline_cv_percent": cv, + "paired_range_percent": [min(degradations), max(degradations)], + } + return {**summary, "rule": verdict_rule(summary, policy)} + + +def pairs_from_samples(samples: list[Any]) -> dict[str, list[dict[str, float]]]: + """Pair a recorded sample list by adjacency, exactly as pairsFromSamples does in JavaScript.""" + measured = [s for s in samples if not (isinstance(s, dict) and s.get("mode") == "warmup")] + if len(measured) % 2: + raise AssertionError("Activity samples do not form whole pairs") + by_mode: dict[str, list[dict[str, float]]] = {} + for i in range(0, len(measured), 2): + first, second = measured[i], measured[i + 1] + if not (isinstance(first, dict) and isinstance(second, dict)): + raise AssertionError("Activity samples must be objects") + off, on = (first, second) if first.get("mode") == "off" else (second, first) + if off.get("mode") != "off" or on.get("mode") == "off" or not isinstance(on.get("mode"), str) or first.get("pair") != second.get("pair"): + raise AssertionError(f"Activity samples {i + 1} and {i + 2} are not one off/on pair") + pairs = by_mode.setdefault(on["mode"], []) + if not _is_int(on.get("pair")) or on["pair"] != len(pairs) or (on["pair"] % 2 == 0) != (first.get("mode") == "off"): + raise AssertionError(f"Activity {on['mode']} pair {on.get('pair')} is out of order") + pairs.append({"off": off.get("tokens_per_second"), "on": on.get("tokens_per_second")}) + return by_mode + + +def check_summary_consistency(item: dict[str, Any], policy: dict[str, Any]) -> None: + """Structure and internal consistency of one recorded mode summary, for evidence without samples.""" + mode = item.get("mode") + status = item.get("status") + if status not in STATUSES: + raise AssertionError(f"{mode} status {status!r} is not a final verdict") + n = item.get("paired_runs") + if not _is_int(n) or n < policy["min_pairs"] or n > policy["max_pairs"] or (n - policy["min_pairs"]) % policy["pair_step"]: + raise AssertionError(f"{mode} paired_runs {n!r} is not a look of the verdict policy") + interval = item.get("interval_percent") + if not (isinstance(interval, list) and len(interval) == 2 and all(_is_number(v) for v in interval) and interval[0] <= interval[1]): + raise AssertionError(f"{mode} interval_percent must be two ordered finite numbers") + median = item.get("median_decode_degradation_percent") + if not _is_number(median) or not interval[0] <= median <= interval[1]: + raise AssertionError(f"{mode} median must be finite and inside its interval") + low, high = interval + expected = "investigate" if low > policy["budget_percent"] else "within-target" if high <= policy["precision_ceiling_percent"] else "inconclusive" + if expected == "inconclusive" and n < policy["max_pairs"]: + raise AssertionError(f"{mode} stopped undecided at {n} pairs, below the {policy['max_pairs']}-pair cap") + if status != expected: + raise AssertionError(f"{mode} status {status} contradicts its interval [{low}, {high}] under the verdict policy") + + +def failure_message(summaries: list[dict[str, Any]]) -> str: + """One line per failing mode: status, point estimate, interval, pair count and the rule that fired.""" + lines = [] + for item in summaries: + status = item.get("status") + if status == "within-target": + continue + kind = {"investigate": "slowdown", "inconclusive": "measurement noise"}.get(status, "unknown verdict") + lines.append(f"{item.get('mode')}: {status} ({kind}): {item.get('rule', 'no rule recorded')}") + return "; ".join(lines) + + +def expected_counts(paired_runs: dict[str, int]) -> tuple[dict[str, int], dict[str, int]]: + """Samples: one warmup, one per pair per mode, and an `off` baseline for every pair. Preflight: + every clients() call for a mode records each of its clients, once before the warmup and once + per pair.""" + samples = {"warmup": 1, "off": sum(paired_runs.values()), **paired_runs} + preflight = {mode: CLIENTS_PER_MODE[mode] * (1 + runs) for mode, runs in paired_runs.items()} + return samples, preflight diff --git a/scripts/webui/activity_performance_verdict_tests.py b/scripts/webui/activity_performance_verdict_tests.py new file mode 100755 index 000000000..92bf38bf1 --- /dev/null +++ b/scripts/webui/activity_performance_verdict_tests.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +# Copyright 2026 Lablup Inc. Licensed under the Apache License, Version 2.0. +"""The Python verdict port against the fixture webui/scripts/activity-performance-verdict.spec.mjs uses.""" +from __future__ import annotations + +import json +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import activity_performance_verdict as verdict # noqa: E402 + +FIXTURE = json.loads(verdict.FIXTURE_PATH.read_text(encoding="utf-8")) +POLICY = verdict.load_policy() + + +def evaluate(case: dict) -> dict: + by_mode = verdict.pairs_from_samples(case["samples"]) + if case["check"] == "pairs": + return {} + if case["check"] == "look": + return {mode: verdict.look_verdict(pairs, POLICY) for mode, pairs in by_mode.items()} + return {mode: verdict.mode_summary(mode, pairs, POLICY) for mode, pairs in by_mode.items()} + + +class FixtureTests(unittest.TestCase): + def test_fixture_was_generated_under_the_policy_in_force(self) -> None: + self.assertEqual(FIXTURE["policy"], POLICY) + + def test_every_fixture_case_reproduces_exactly(self) -> None: + for case in FIXTURE["cases"]: + with self.subTest(case["name"]): + expected = case["expected"] + if "error" in expected: + with self.assertRaisesRegex(AssertionError, expected["error"]): + evaluate(case) + continue + result = evaluate(case) + self.assertEqual(sorted(result), sorted(expected)) + for mode, want in expected.items(): + self.assertEqual({key: result[mode][key] for key in want}, want) + + def test_no_recorded_noise_failure_is_a_slowdown(self) -> None: + recorded = [case for case in FIXTURE["cases"] if case["name"].startswith("recorded ")] + self.assertGreaterEqual(len(recorded), 10) + for case in recorded: + for want in case["expected"].values(): + self.assertNotEqual(want["status"], "investigate") + + def test_fixture_covers_every_final_status(self) -> None: + statuses = {want.get("status") for case in FIXTURE["cases"] for want in case["expected"].values() if isinstance(want, dict)} + self.assertTrue({"investigate", "within-target", "inconclusive", "continue"} <= statuses) + + +class RuleTests(unittest.TestCase): + def test_sign_test_rank_matches_the_exact_binomial_tail(self) -> None: + self.assertEqual([verdict.sign_test_rank(n, 0.01) for n in (6, 7, 12, 40)], [0, 1, 2, 13]) + self.assertEqual(verdict.median_interval([5, 1, 4, 2, 3, 6, 7, 8, 9, 10, 11, 12], POLICY), (6.5, [2, 11])) + self.assertEqual(verdict.median_interval([1, 2, 3], POLICY), (2, None)) + + def test_policy_validation_rejects_unsafe_or_incomplete_policies(self) -> None: + for change in ({"tail_alpha": 0}, {"tail_alpha": 0.5}, {"min_pairs": 6}, {"pair_step": 1}, {"max_pairs": 52}, {"max_pairs": 41}, {"precision_ceiling_percent": 1}, {"budget_percent": float("nan")}, {"extra": 1}, {"min_pairs": True}): + with self.subTest(change), self.assertRaisesRegex(AssertionError, "policy"): + verdict.validate_policy({**POLICY, **change}) + missing = dict(POLICY) + del missing["max_pairs"] + with self.assertRaisesRegex(AssertionError, "policy"): + verdict.validate_policy(missing) + + def test_summary_consistency_follows_the_policy(self) -> None: + good = {"mode": "one-visible", "status": "within-target", "paired_runs": 12, "median_decode_degradation_percent": 0.5, "interval_percent": [-1.0, 3.0]} + verdict.check_summary_consistency(good, POLICY) + cases = [ + ({"status": "investigate"}, "contradicts"), + ({"interval_percent": [-1.0, 4.5]}, "below the 40-pair cap"), + ({"interval_percent": [2.5, 4.5], "median_decode_degradation_percent": 3.0}, "contradicts"), + ({"paired_runs": 13}, "not a look"), + ({"paired_runs": 42}, "not a look"), + ({"median_decode_degradation_percent": 5.0}, "inside its interval"), + ({"interval_percent": [3.0, 1.0]}, "ordered"), + ({"status": "continue"}, "final verdict"), + ] + for change, message in cases: + with self.subTest(change), self.assertRaisesRegex(AssertionError, message): + verdict.check_summary_consistency({**good, **change}, POLICY) + verdict.check_summary_consistency({**good, "status": "inconclusive", "paired_runs": 40, "interval_percent": [-1.0, 4.5]}, POLICY) + verdict.check_summary_consistency({**good, "status": "investigate", "interval_percent": [2.5, 4.5], "median_decode_degradation_percent": 3.0}, POLICY) + + def test_failure_message_names_status_kind_and_rule(self) -> None: + message = verdict.failure_message([ + {"mode": "one-visible", "status": "within-target", "rule": "fine"}, + {"mode": "two-visible", "status": "investigate", "rule": "slow rule"}, + {"mode": "hidden", "status": "inconclusive", "rule": "noisy rule"}, + ]) + self.assertEqual(message, "two-visible: investigate (slowdown): slow rule; hidden: inconclusive (measurement noise): noisy rule") + + def test_expected_counts_follow_paired_runs(self) -> None: + self.assertEqual(verdict.expected_counts({"one-visible": 12, "two-visible": 14}), ({"warmup": 1, "off": 26, "one-visible": 12, "two-visible": 14}, {"one-visible": 13, "two-visible": 30})) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/webui/summarize_activity_evidence.py b/scripts/webui/summarize_activity_evidence.py index e5c789046..7f642c385 100755 --- a/scripts/webui/summarize_activity_evidence.py +++ b/scripts/webui/summarize_activity_evidence.py @@ -10,6 +10,10 @@ import argparse, json, math, os, re, stat from pathlib import Path from typing import Any +import sys + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import activity_performance_verdict as verdict # noqa: E402 FORBIDDEN_KEYS = {"path", "work_dir", "model_view", "output", "log", "processes", "env"} HEX64 = re.compile(r"^[0-9a-f]{64}$") @@ -54,6 +58,11 @@ def activity_summary(records: list[Any], hidden_native: str = "measured") -> dic require(summary.get("hidden_native") == hidden_native, f"hidden_native must be {hidden_native}") summaries = summary.get("summaries") require(isinstance(summaries, list) and summaries, "mode summaries missing") + # This record carries no samples, so the verdict cannot be re-derived here (the verifier did + # that). What is checked is that each summary is internally consistent with the repository's + # verdict policy, which keeps the thresholds in one file shared with the harness (#1925). + policy = verdict.load_policy() + require(summary.get("verdict_policy") == policy, "activity summary must carry the repository's verdict policy") allowed = [] seen: set[str] = set() for item in summaries: @@ -61,12 +70,10 @@ def activity_summary(records: list[Any], hidden_native: str = "measured") -> dic mode = item.get("mode") require(mode in expected_modes and mode not in seen, "mode summaries must contain each required mode once") seen.add(mode) - require(item.get("status") == "within-target", f"{mode} status must be within-target") - require(item.get("paired_runs") == 5, f"{mode} paired_runs must be 5") - degradation = finite_number(item.get("median_decode_degradation_percent"), f"{mode} degradation") + verdict.check_summary_consistency(item, policy) + require(item.get("status") == "within-target", f"{mode} status must be within-target: {verdict.failure_message([item])}") cv = finite_number(item.get("baseline_cv_percent"), f"{mode} baseline cv") - require(degradation <= 2, f"{mode} degradation exceeds budget") - require(0 <= cv <= 5, f"{mode} baseline cv exceeds budget") + require(cv >= 0, f"{mode} baseline cv must not be negative") pair_range = item.get("paired_range_percent") require(isinstance(pair_range, list) and len(pair_range) == 2, f"{mode} paired_range_percent must have two values") low = finite_number(pair_range[0], f"{mode} paired range low") @@ -75,18 +82,21 @@ def activity_summary(records: list[Any], hidden_native: str = "measured") -> dic allowed.append({ "mode": mode, "status": "within-target", - "paired_runs": 5, - "median_decode_degradation_percent": degradation, + "paired_runs": item["paired_runs"], + "median_decode_degradation_percent": item["median_decode_degradation_percent"], + "interval_percent": list(item["interval_percent"]), "baseline_cv_percent": cv, "paired_range_percent": [low, high], }) require(seen == expected_modes, f"mode summaries missing required modes: {sorted(expected_modes - seen)}") - out = {"status": summary["status"], "hidden_native": summary["hidden_native"], "output_sha256": output_sha256, "summaries": allowed} + out = {"status": summary["status"], "hidden_native": summary["hidden_native"], "output_sha256": output_sha256, "verdict_policy": policy, "summaries": allowed} + expected_counts = dict(zip(("sample_counts", "preflight_counts"), verdict.expected_counts({item["mode"]: item["paired_runs"] for item in allowed}))) for key in ("sample_counts", "preflight_counts"): if key in records[0]: value = records[0][key] require(isinstance(value, dict), f"{key} must be an object") out[key] = {str(k): nonnegative_int(v, f"{key}.{k}") for k, v in value.items()} + require(out[key] == expected_counts[key], f"{key} {out[key]} do not match the summaries' paired_runs: want {expected_counts[key]}") return out def checkpoint_summary(raw: Any) -> dict[str, Any]: diff --git a/scripts/webui/summarize_activity_evidence_tests.py b/scripts/webui/summarize_activity_evidence_tests.py index 76cb0d5d0..f188649cf 100755 --- a/scripts/webui/summarize_activity_evidence_tests.py +++ b/scripts/webui/summarize_activity_evidence_tests.py @@ -9,6 +9,11 @@ summarize = importlib.util.module_from_spec(spec) sys.modules["summarize_activity_evidence"] = summarize spec.loader.exec_module(summarize) +verdict = sys.modules["activity_performance_verdict"] +POLICY = verdict.load_policy() + +def mode_summary(mode: str) -> dict[str, object]: + return {"mode": mode, "status": "within-target", "paired_runs": 12, "median_decode_degradation_percent": 1.0, "interval_percent": [-1.0, 3.0], "baseline_cv_percent": 1.0, "paired_range_percent": [-2.0, 4.0], "rule": "median 1.00%, 98% interval [-1.00%, 3.00%] over 12 pairs; within target"} def full_evidence() -> dict[str, object]: return { @@ -36,10 +41,8 @@ def full_evidence() -> dict[str, object]: "summary": { "status": "within-target", "hidden_native": "measured", - "summaries": [ - {"mode": mode, "status": "within-target", "paired_runs": 5, "median_decode_degradation_percent": 1.0, "baseline_cv_percent": 1.0, "paired_range_percent": [0.0, 1.0]} - for mode in ("one-visible", "two-visible", "hidden") - ], + "verdict_policy": POLICY, + "summaries": [mode_summary(mode) for mode in ("one-visible", "two-visible", "hidden")], }, }], "server_shutdown": {"exit_code": 0, "forced": False, "process_group_empty": True, "log": "/secret/server.log"}, @@ -51,10 +54,8 @@ def visible_only_evidence() -> dict[str, object]: record["summary"] = { "status": "incomplete", "hidden_native": "not-run", - "summaries": [ - {"mode": mode, "status": "within-target", "paired_runs": 5, "median_decode_degradation_percent": 1.0, "baseline_cv_percent": 1.0, "paired_range_percent": [0.0, 1.0]} - for mode in ("one-visible", "two-visible") - ], + "verdict_policy": POLICY, + "summaries": [mode_summary(mode) for mode in ("one-visible", "two-visible")], } return data @@ -71,15 +72,44 @@ def test_declared_deferral_accepts_visible_only_and_default_still_rejects_it(sel with self.assertRaises(AssertionError): summarize.build_summary(full_evidence(), "not-run") - def test_declared_deferral_still_enforces_every_numeric_budget(self) -> None: - over_budget = visible_only_evidence() - over_budget["activity_performance"][0]["summary"]["summaries"][0]["median_decode_degradation_percent"] = 3.0 # type: ignore[index] - with self.assertRaises(AssertionError): - summarize.build_summary(over_budget, "not-run") + def test_declared_deferral_still_enforces_the_verdict(self) -> None: + summaries = lambda data: data["activity_performance"][0]["summary"]["summaries"] # noqa: E731 + slowdown = visible_only_evidence() + summaries(slowdown)[0].update({"median_decode_degradation_percent": 3.0, "interval_percent": [2.5, 3.5]}) + with self.assertRaisesRegex(AssertionError, "contradicts its interval"): + summarize.build_summary(slowdown, "not-run") + summaries(slowdown)[0]["status"] = "investigate" + with self.assertRaisesRegex(AssertionError, "investigate \\(slowdown\\)"): + summarize.build_summary(slowdown, "not-run") + noise = visible_only_evidence() + summaries(noise)[1].update({"status": "inconclusive", "paired_runs": 40, "interval_percent": [-1.0, 4.5]}) + with self.assertRaisesRegex(AssertionError, "inconclusive \\(measurement noise\\)"): + summarize.build_summary(noise, "not-run") + stale = visible_only_evidence() + stale["activity_performance"][0]["summary"]["verdict_policy"] = {**POLICY, "budget_percent": 5} # type: ignore[index] + with self.assertRaisesRegex(AssertionError, "verdict policy"): + summarize.build_summary(stale, "not-run") + old_shape = visible_only_evidence() + summaries(old_shape)[0]["paired_runs"] = 5 + with self.assertRaisesRegex(AssertionError, "not a look"): + summarize.build_summary(old_shape, "not-run") + + def test_baseline_cv_is_a_diagnostic_and_counts_follow_paired_runs(self) -> None: noisy = visible_only_evidence() noisy["activity_performance"][0]["summary"]["summaries"][1]["baseline_cv_percent"] = 9.0 # type: ignore[index] - with self.assertRaises(AssertionError): - summarize.build_summary(noisy, "not-run") + result = summarize.build_summary(noisy, "not-run") + self.assertEqual(result["activity_performance"]["summaries"][1]["baseline_cv_percent"], 9.0) + self.assertEqual(result["activity_performance"]["summaries"][1]["interval_percent"], [-1.0, 3.0]) + self.assertEqual(result["activity_performance"]["verdict_policy"], POLICY) + self.assertNotIn("rule", result["activity_performance"]["summaries"][1]) + counted = visible_only_evidence() + record = counted["activity_performance"][0] # type: ignore[index] + record["sample_counts"] = {"warmup": 1, "off": 24, "one-visible": 12, "two-visible": 12} + record["preflight_counts"] = {"one-visible": 13, "two-visible": 26} + self.assertEqual(summarize.build_summary(counted, "not-run")["activity_performance"]["sample_counts"]["off"], 24) + record["sample_counts"] = {"warmup": 1, "off": 10, "one-visible": 5, "two-visible": 5} + with self.assertRaisesRegex(AssertionError, "sample_counts"): + summarize.build_summary(counted, "not-run") def test_summary_allowlist_excludes_paths_and_keeps_native_hidden_status(self) -> None: result = summarize.build_summary(full_evidence()) diff --git a/scripts/webui/verify_activity_performance.py b/scripts/webui/verify_activity_performance.py index 9d680e7d3..be43af009 100755 --- a/scripts/webui/verify_activity_performance.py +++ b/scripts/webui/verify_activity_performance.py @@ -30,18 +30,15 @@ from pathlib import Path from typing import Any, Iterable +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import activity_performance_verdict as verdict # noqa: E402 + MAX_BODY = 16 * 1024 * 1024 MODEL_ID_RE = re.compile(r"^mdl_[A-Za-z0-9_-]{43}$") HEX40_RE = re.compile(r"^[0-9a-f]{40}$") SECRET_RE = re.compile(r"(Bearer\s+)[A-Za-z0-9._~+/=-]+") REQUIRED_MODES = {"one-visible", "two-visible", "hidden"} VISIBLE_ONLY_MODES = {"one-visible", "two-visible"} -EXPECTED_SAMPLE_COUNTS = {"warmup": 1, "off": 15, "one-visible": 5, "two-visible": 5, "hidden": 5} -EXPECTED_PREFLIGHT_COUNTS = {"one-visible": 6, "two-visible": 12, "hidden": 6} -# A visible-only run drops the hidden mode: its five paired runs and the six preflight records -# that belong to it, and the five "off" baselines they were paired against. -VISIBLE_ONLY_SAMPLE_COUNTS = {"warmup": 1, "off": 10, "one-visible": 5, "two-visible": 5} -VISIBLE_ONLY_PREFLIGHT_COUNTS = {"one-visible": 6, "two-visible": 12} @dataclass class OwnedProcess: @@ -236,28 +233,34 @@ def materialize_model_view(source: Path, work: Path) -> tuple[Path, str]: raise AssertionError(f"failed to create isolated model view: {'; '.join(errors)}") def validate_activity_output(path: Path, perf_mode: str = "full") -> dict[str, Any]: - # A visible-only run is a declared deferral of the hidden acceptance, not a weaker pass: every - # numeric budget below still applies, and the expected mode, sample and preflight counts move - # to the visible-only set so a run that silently dropped a mode is still caught. + # A visible-only run is a declared deferral of the hidden acceptance, not a weaker pass: the + # verdict below still applies, and the expected modes move to the visible-only set so a run + # that silently dropped a mode is still caught. hidden_deferred = perf_mode != "full" expected_status = "incomplete" if hidden_deferred else "within-target" expected_hidden_native = "not-run" if hidden_deferred else "measured" expected_modes = VISIBLE_ONLY_MODES if hidden_deferred else REQUIRED_MODES - expected_preflight = VISIBLE_ONLY_PREFLIGHT_COUNTS if hidden_deferred else EXPECTED_PREFLIGHT_COUNTS - expected_samples = VISIBLE_ONLY_SAMPLE_COUNTS if hidden_deferred else EXPECTED_SAMPLE_COUNTS data = json.loads(path.read_text(encoding="utf-8")) if not isinstance(data, dict): raise AssertionError("activity performance output must be a JSON object") - if data.get("status") != expected_status: - raise AssertionError(f"activity performance status is {data.get('status')!r}, want {expected_status!r}") - if data.get("hidden_native") != expected_hidden_native: - raise AssertionError(f"hidden_native is {data.get('hidden_native')!r}, want {expected_hidden_native!r}") summaries = data.get("summaries") - if not isinstance(summaries, list) or not summaries: + if not isinstance(summaries, list) or not summaries or not all(isinstance(item, dict) for item in summaries): raise AssertionError("activity performance output must include summaries") - bad = [item for item in summaries if not isinstance(item, dict) or item.get("status") != "within-target"] - if bad: - raise AssertionError(f"activity performance has non-target summaries: {bad}") + # The verdict is re-derived from the raw samples with the Python port of the harness's rule, + # under the repository's policy file, and must match what the harness recorded (#1925). + policy = verdict.load_policy() + if data.get("verdict_policy") != policy: + raise AssertionError(f"activity performance used verdict policy {data.get('verdict_policy')!r}, want the repository's {policy!r}") + samples = data.get("samples") + if not isinstance(samples, list): + raise AssertionError("activity performance output must include samples") + for item in samples: + if not isinstance(item, dict) or not isinstance(item.get("mode"), str): + raise AssertionError("sample records must include mode") + finite_number(item.get("tokens_per_second"), f"{item['mode']} tokens_per_second", positive=True) + finite_number(item.get("predicted_tokens"), f"{item['mode']} predicted_tokens", positive=True) + finite_number(item.get("predicted_ms"), f"{item['mode']} predicted_ms", positive=True) + pairs = verdict.pairs_from_samples(samples) seen_summary_modes: set[str] = set() for item in summaries: mode = item.get("mode") @@ -266,24 +269,29 @@ def validate_activity_output(path: Path, perf_mode: str = "full") -> dict[str, A if mode in seen_summary_modes: raise AssertionError(f"duplicate summary mode: {mode}") seen_summary_modes.add(mode) - if item.get("paired_runs") != 5: - raise AssertionError(f"{mode} summary must have paired_runs=5") - degradation = finite_number(item.get("median_decode_degradation_percent"), f"{mode} degradation") - cv = finite_number(item.get("baseline_cv_percent"), f"{mode} baseline CV") - if cv < 0: - raise AssertionError(f"{mode} baseline CV must not be negative") - if degradation > 2 or cv > 5: - raise AssertionError(f"{mode} summary exceeded targets despite within-target status") - pair_range = item.get("paired_range_percent") - if not isinstance(pair_range, list) or len(pair_range) != 2: - raise AssertionError(f"{mode} paired_range_percent must have two entries") - range_min = finite_number(pair_range[0], f"{mode} paired range min") - range_max = finite_number(pair_range[1], f"{mode} paired range max") - if range_min > range_max: - raise AssertionError(f"{mode} paired_range_percent must be ordered") + derived = verdict.mode_summary(mode, pairs.get(mode, []), policy) + for key in ("status", "paired_runs", "median_decode_degradation_percent", "interval_percent"): + if item.get(key) != derived[key]: + raise AssertionError(f"{mode} {key} is {item.get(key)!r}, but its samples give {derived[key]!r}") + for key in ("baseline_cv_percent", "paired_range_percent"): + recorded = item.get(key) + values = recorded if isinstance(recorded, list) else [recorded] + expected = derived[key] if isinstance(derived[key], list) else [derived[key]] + if len(values) != len(expected) or not all(isinstance(v, (int, float)) and not isinstance(v, bool) and abs(v - e) <= 1e-9 * max(1.0, abs(e)) for v, e in zip(values, expected)): + raise AssertionError(f"{mode} {key} is {recorded!r}, but its samples give {derived[key]!r}") missing = expected_modes - seen_summary_modes if missing: raise AssertionError(f"activity performance missing mode summaries: {sorted(missing)}") + if set(pairs) != seen_summary_modes: + raise AssertionError(f"activity performance samples cover modes {sorted(pairs)}, summaries {sorted(seen_summary_modes)}") + failing = verdict.failure_message(summaries) + if failing: + raise AssertionError(f"Activity performance gate: {failing}") + if data.get("status") != expected_status: + raise AssertionError(f"activity performance status is {data.get('status')!r}, want {expected_status!r}") + if data.get("hidden_native") != expected_hidden_native: + raise AssertionError(f"hidden_native is {data.get('hidden_native')!r}, want {expected_hidden_native!r}") + expected_samples, expected_preflight = verdict.expected_counts({item["mode"]: item["paired_runs"] for item in summaries}) preflight = data.get("preflight") if not isinstance(preflight, list) or not preflight: raise AssertionError("activity performance output must include preflight geometry/visibility records") @@ -298,20 +306,12 @@ def validate_activity_output(path: Path, perf_mode: str = "full") -> dict[str, A finite_number(geometry.get(field), f"preflight {field}", positive=True) preflight_counts[item["mode"]] = preflight_counts.get(item["mode"], 0) + 1 if preflight_counts != expected_preflight: - raise AssertionError(f"unexpected preflight counts: {preflight_counts}") - samples = data.get("samples") - if not isinstance(samples, list): - raise AssertionError("activity performance output must include samples") + raise AssertionError(f"unexpected preflight counts: {preflight_counts}, want {expected_preflight}") sample_counts: dict[str, int] = {} for item in samples: - if not isinstance(item, dict) or not isinstance(item.get("mode"), str): - raise AssertionError("sample records must include mode") sample_counts[item["mode"]] = sample_counts.get(item["mode"], 0) + 1 - finite_number(item.get("tokens_per_second"), f"{item['mode']} tokens_per_second", positive=True) - finite_number(item.get("predicted_tokens"), f"{item['mode']} predicted_tokens", positive=True) - finite_number(item.get("predicted_ms"), f"{item['mode']} predicted_ms", positive=True) if sample_counts != expected_samples: - raise AssertionError(f"unexpected sample counts: {sample_counts}") + raise AssertionError(f"unexpected sample counts: {sample_counts}, want {expected_samples}") return data def parse_features(raw: str) -> list[str]: @@ -751,7 +751,7 @@ def run(args: argparse.Namespace) -> dict[str, Any]: runtime_loaded = runtime_snapshot(h.base_prefixed, key, identity["model_id"]) h.record("runtime_observations", {"stage": "loaded", "entry": loaded, "runtime": runtime_loaded}) activity, activity_output, activity_log = run_activity_script(h, identity["model_id"], identity["inference_id"]) - h.record("activity_performance", {"status": "passed", "output": str(activity_output), "output_sha256": sha256(activity_output), "log": str(activity_log), "summary": {"status": activity.get("status"), "hidden_native": activity.get("hidden_native"), "summaries": activity.get("summaries")}}) + h.record("activity_performance", {"status": "passed", "output": str(activity_output), "output_sha256": sha256(activity_output), "log": str(activity_log), "summary": {"status": activity.get("status"), "hidden_native": activity.get("hidden_native"), "verdict_policy": activity.get("verdict_policy"), "summaries": activity.get("summaries")}}) latest = wait_entry(h.base_prefixed, key, identity["model_id"], lambda item: isinstance(item.get("identity"), dict), "latest revision before unload", timeout=30) latest_identity = entry_identity(latest) unload = unload_model(h.base_prefixed, key, identity["model_id"], latest_identity["revision"]) diff --git a/scripts/webui/verify_activity_performance_tests.py b/scripts/webui/verify_activity_performance_tests.py index 93ebec044..8348cacac 100755 --- a/scripts/webui/verify_activity_performance_tests.py +++ b/scripts/webui/verify_activity_performance_tests.py @@ -23,44 +23,46 @@ spec.loader.exec_module(verify) +verdict = sys.modules["activity_performance_verdict"] +POLICY = verdict.load_policy() +FIXTURE = json.loads(verdict.FIXTURE_PATH.read_text(encoding="utf-8")) +GEOMETRY = {"innerWidth": 700, "innerHeight": 900, "visualWidth": 700, "visualHeight": 900} + + +def fixture_pairs(prefix: str) -> list[dict[str, float]]: + case = next(item for item in FIXTURE["cases"] if item["name"].startswith(prefix)) + (pairs,) = verdict.pairs_from_samples(case["samples"]).values() + return pairs + + +NULL_PAIRS = fixture_pairs("null: every recorded one-visible") + + +def activity_output(modes: tuple[str, ...], pairs_by_mode: dict[str, list[dict[str, float]]] | None = None) -> dict[str, object]: + """Harness output shaped exactly as activity-performance.mjs writes it, for recorded pairs.""" + samples: list[dict[str, object]] = [{"mode": "warmup", "pair": -1, "tokens_per_second": 150.0, "predicted_tokens": 256, "predicted_ms": 1700}] + preflight = [{"mode": mode, "client": 0, "geometry": GEOMETRY} for mode in modes for _ in range(verdict.CLIENTS_PER_MODE[mode])] + summaries = [] + for mode in modes: + pairs = (pairs_by_mode or {}).get(mode, NULL_PAIRS) + for index, pair in enumerate(pairs): + off = {"mode": "off", "pair": index, "tokens_per_second": pair["off"], "predicted_tokens": 256, "predicted_ms": 1700} + on = {"mode": mode, "pair": index, "tokens_per_second": pair["on"], "predicted_tokens": 256, "predicted_ms": 1700} + samples.extend([off, on] if index % 2 == 0 else [on, off]) + preflight.extend({"mode": mode, "client": client, "geometry": GEOMETRY} for client in range(verdict.CLIENTS_PER_MODE[mode])) + summaries.append(verdict.mode_summary(mode, pairs, POLICY)) + hidden_ran = "hidden" in modes + failing = {item["status"] for item in summaries} - {"within-target"} + status = "incomplete" if not hidden_ran else "investigate" if "investigate" in failing else "inconclusive" if failing else "within-target" + return {"status": status, "hidden_native": "measured" if hidden_ran else "not-run", "preflight": preflight, "verdict_policy": POLICY, "samples": samples, "summaries": summaries} + + def valid_activity_output() -> dict[str, object]: - samples = [{"mode": "warmup", "tokens_per_second": 1.0, "predicted_tokens": 1, "predicted_ms": 1.0}] - for mode, count in {"off": 15, "one-visible": 5, "two-visible": 5, "hidden": 5}.items(): - samples.extend({"mode": mode, "tokens_per_second": 1.0, "predicted_tokens": 1, "predicted_ms": 1.0} for _ in range(count)) - geom = {"innerWidth": 700, "innerHeight": 900, "visualWidth": 700, "visualHeight": 900} - preflight = [] - for mode, count in {"one-visible": 6, "two-visible": 12, "hidden": 6}.items(): - preflight.extend({"mode": mode, "geometry": geom} for _ in range(count)) - return { - "status": "within-target", - "hidden_native": "measured", - "preflight": preflight, - "samples": samples, - "summaries": [ - {"mode": mode, "paired_runs": 5, "median_decode_degradation_percent": 1.0, "baseline_cv_percent": 1.0, "paired_range_percent": [0.0, 1.0], "status": "within-target"} - for mode in ("one-visible", "two-visible", "hidden") - ], - } + return activity_output(("one-visible", "two-visible", "hidden")) def visible_only_activity_output() -> dict[str, object]: - samples = [{"mode": "warmup", "tokens_per_second": 1.0, "predicted_tokens": 1, "predicted_ms": 1.0}] - for mode, count in {"off": 10, "one-visible": 5, "two-visible": 5}.items(): - samples.extend({"mode": mode, "tokens_per_second": 1.0, "predicted_tokens": 1, "predicted_ms": 1.0} for _ in range(count)) - geom = {"innerWidth": 700, "innerHeight": 900, "visualWidth": 700, "visualHeight": 900} - preflight = [] - for mode, count in {"one-visible": 6, "two-visible": 12}.items(): - preflight.extend({"mode": mode, "geometry": geom} for _ in range(count)) - return { - "status": "incomplete", - "hidden_native": "not-run", - "preflight": preflight, - "samples": samples, - "summaries": [ - {"mode": mode, "paired_runs": 5, "median_decode_degradation_percent": 1.0, "baseline_cv_percent": 1.0, "paired_range_percent": [0.0, 1.0], "status": "within-target"} - for mode in ("one-visible", "two-visible") - ], - } + return activity_output(("one-visible", "two-visible")) class ActivityPerformanceHelperTests(unittest.TestCase): @@ -382,6 +384,52 @@ def test_validate_activity_output_rejects_nonfinite_or_wrong_counts(self) -> Non with self.assertRaises(AssertionError): verify.validate_activity_output(path) + def test_validate_activity_output_reports_slowdown_and_noise_distinctly(self) -> None: + # The job log has to tell a slowdown from measurement noise without the artifact (#1925). + slow = fixture_pairs("injected 6% slowdown on the recorded two-visible") + noisy = fixture_pairs("noise too wide to decide") + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "activity.json" + path.write_text(json.dumps(activity_output(("one-visible", "two-visible"), {"two-visible": slow}))) + with self.assertRaisesRegex(AssertionError, r"two-visible: investigate \(slowdown\): median 6\.12%, 98% interval \[2\.15%, 8\.57%\] over 14 pairs; slowdown: the interval lies wholly above the 2% budget"): + verify.validate_activity_output(path, "visible-only-headed") + path.write_text(json.dumps(activity_output(("one-visible", "two-visible"), {"one-visible": noisy}))) + with self.assertRaisesRegex(AssertionError, r"one-visible: inconclusive \(measurement noise\): .* over 40 pairs; measurement noise, not a slowdown"): + verify.validate_activity_output(path, "visible-only-headed") + + def test_baseline_cv_alone_no_longer_fails(self) -> None: + drift = fixture_pairs("baseline drift with CV above 5%") + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "activity.json" + data = activity_output(("one-visible", "two-visible"), {"two-visible": drift}) + self.assertGreater(data["summaries"][1]["baseline_cv_percent"], 5) # type: ignore[index] + path.write_text(json.dumps(data)) + self.assertEqual(verify.validate_activity_output(path, "visible-only-headed")["summaries"][1]["status"], "within-target") + + def test_validate_activity_output_rederives_the_verdict_from_samples(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "activity.json" + data = visible_only_activity_output() + data["verdict_policy"] = {**POLICY, "precision_ceiling_percent": 10} + path.write_text(json.dumps(data)) + with self.assertRaisesRegex(AssertionError, "verdict policy"): + verify.validate_activity_output(path, "visible-only-headed") + data = activity_output(("one-visible", "two-visible"), {"two-visible": fixture_pairs("injected 6% slowdown on the recorded two-visible")}) + data["summaries"][1]["status"] = "within-target" # type: ignore[index] + path.write_text(json.dumps(data)) + with self.assertRaisesRegex(AssertionError, "status is 'within-target', but its samples give 'investigate'"): + verify.validate_activity_output(path, "visible-only-headed") + data = visible_only_activity_output() + data["preflight"] = data["preflight"][:-1] # type: ignore[index] + path.write_text(json.dumps(data)) + with self.assertRaisesRegex(AssertionError, "preflight counts"): + verify.validate_activity_output(path, "visible-only-headed") + data = visible_only_activity_output() + data["samples"].append({"mode": "warmup", "pair": -1, "tokens_per_second": 1.0, "predicted_tokens": 1, "predicted_ms": 1.0}) # type: ignore[union-attr] + path.write_text(json.dumps(data)) + with self.assertRaisesRegex(AssertionError, "sample counts"): + verify.validate_activity_output(path, "visible-only-headed") + if __name__ == "__main__": unittest.main() diff --git a/webui/scripts/activity-performance-config.mjs b/webui/scripts/activity-performance-config.mjs index 4c28bde61..d2323fa51 100644 --- a/webui/scripts/activity-performance-config.mjs +++ b/webui/scripts/activity-performance-config.mjs @@ -1,5 +1,8 @@ // Copyright 2026 Lablup Inc. Licensed under the Apache License, Version 2.0. +/* global URL */ // Pure acceptance policy: diagnostic evidence must never become a full pass. +import { readFileSync } from 'node:fs'; + export function performanceMode(value = 'full') { if (value === 'full') return { name: value, headless: false, modes: ['one-visible', 'two-visible', 'hidden'] }; if (value === 'visible-only-headless') return { name: value, headless: true, modes: ['one-visible', 'two-visible'] }; @@ -15,14 +18,126 @@ export function assertGeometry(value) { } export function performanceCompletion(mode, summaries) { // Completeness follows whether the hidden acceptance actually ran, not whether the browser was - // headed: a headed run that skips the hidden mode is still an incomplete acceptance. + // headed: a headed run that skips the hidden mode is still an incomplete acceptance. A measured + // slowdown outranks an undecided measurement, so `investigate` wins over `inconclusive`. const hiddenRan = mode.modes.includes('hidden'); + const has = (status) => summaries.some((x) => x.status === status); return { - status: !hiddenRan ? 'incomplete' : summaries.some((x) => x.status === 'investigate') ? 'investigate' : 'within-target', + status: !hiddenRan ? 'incomplete' : has('investigate') ? 'investigate' : has('inconclusive') ? 'inconclusive' : 'within-target', scope: mode.name, hidden_native: hiddenRan ? 'measured' : 'not-run', limitation: hiddenRan ? null : mode.headless ? 'Visible headless diagnostic only; native hidden-tab and interactive host acceptance remain outstanding.' - : 'Headed visible-mode observation overhead only; native hidden acceptance is not run because the automation browser reports document.hidden as false for a window it reports as minimized.', + : 'Headed visible-mode overhead only; native hidden acceptance is not run because the automation browser reports document.hidden as false for a window it reports as minimized.', }; } + +// Decode-overhead verdict (#1925). Every constant lives in activity-performance-policy.json, which +// scripts/webui/activity_performance_verdict.py reads too, so the harness and both validators judge +// one rule. Both implementations are checked against activity-performance-fixtures.json. +// +// Each pair measures the same request with the Activity clients off and on, in alternating order, +// and scores (off - on) / off * 100: positive is a slowdown. Per-pair noise on the GB10 runner is +// heavy-tailed (kurtosis above 10 across 636 recorded runs), so the point estimate is the median +// and its interval is the distribution-free sign-test interval [x(k), x(n+1-k)], which needs no +// variance estimate and no bootstrap randomness. Sampling starts at min_pairs and adds pair_step +// pairs per look until one of: +// investigate - the interval lies wholly above budget_percent: evidence of a slowdown. +// within-target - the interval does not lie above the budget and its upper bound is at most +// precision_ceiling_percent, so the run was precise enough to have caught a +// slowdown well past the budget. +// inconclusive - max_pairs reached with neither: measurement noise, not a slowdown. +// Baseline CV and the paired range are diagnostics only; neither decides anything. +const policyKeys = ['budget_percent', 'precision_ceiling_percent', 'tail_alpha', 'min_pairs', 'pair_step', 'max_pairs']; +export function validatePolicy(value) { + if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error('Activity verdict policy must be an object.'); + const keys = Object.keys(value).sort(); + if (keys.join() !== [...policyKeys].sort().join()) throw new Error(`Activity verdict policy must have exactly: ${policyKeys.join(', ')}.`); + const { budget_percent: budget, precision_ceiling_percent: ceiling, tail_alpha: alpha, min_pairs: min, pair_step: step, max_pairs: max } = value; + if (![budget, ceiling, alpha].every(Number.isFinite) || !(ceiling >= budget) || !(alpha > 0 && alpha < 0.5)) throw new Error('Activity verdict policy thresholds are invalid.'); + // Binomial sums stay exact in a double up to n = 50, which keeps JavaScript and Python identical. + if (![min, step, max].every(Number.isInteger) || step < 2 || step % 2 !== 0 || min % 2 !== 0 || max > 50 || max < min || (max - min) % step !== 0) throw new Error('Activity verdict policy pair counts are invalid.'); + if (signTestRank(min, alpha) < 1) throw new Error('Activity verdict policy min_pairs is too small for its tail_alpha.'); + return Object.freeze({ ...value }); +} +export const verdictPolicy = validatePolicy(JSON.parse(readFileSync(new URL('./activity-performance-policy.json', import.meta.url), 'utf8'))); + +// Largest k with P(Binomial(n, 1/2) <= k - 1) <= alpha; 0 when no order statistic qualifies. +export function signTestRank(n, alpha) { + let choose = 1; let cumulative = 0; let k = 0; + for (let j = 0; j <= n; j++) { + cumulative += choose; + if (cumulative > alpha * 2 ** n) break; + k = j + 1; + choose = choose * (n - j) / (j + 1); + } + return k; +} +export function pairDegradation(pair) { + if (!(Number.isFinite(pair.off) && pair.off > 0 && Number.isFinite(pair.on) && pair.on > 0)) throw new Error('Activity pair rates must be finite and positive.'); + return (pair.off - pair.on) / pair.off * 100; +} +export function medianInterval(values, policy = verdictPolicy) { + const sorted = [...values].sort((a, b) => a - b); + const n = sorted.length; + if (n === 0) throw new Error('Activity verdict needs at least one pair.'); + const median = n % 2 === 1 ? sorted[(n - 1) / 2] : (sorted[n / 2 - 1] + sorted[n / 2]) / 2; + const k = signTestRank(n, policy.tail_alpha); + return { median, interval: k < 1 ? null : [sorted[k - 1], sorted[n - k]] }; +} +// The verdict for exactly these pairs at one look. `continue` means: not a look, or undecided below max_pairs. +export function lookVerdict(pairs, policy = verdictPolicy) { + const n = pairs.length; + const { median, interval } = medianInterval(pairs.map(pairDegradation), policy); + const base = { paired_runs: n, median_decode_degradation_percent: median, interval_percent: interval }; + if (n < policy.min_pairs || n > policy.max_pairs || (n - policy.min_pairs) % policy.pair_step !== 0) return { ...base, status: 'continue' }; + if (interval[0] > policy.budget_percent) return { ...base, status: 'investigate' }; + if (interval[1] <= policy.precision_ceiling_percent) return { ...base, status: 'within-target' }; + return { ...base, status: n >= policy.max_pairs ? 'inconclusive' : 'continue' }; +} +// Replays every look over a recorded sequence; the harness must have stopped at the first decision. +export function sequentialVerdict(pairs, policy = verdictPolicy) { + for (let n = policy.min_pairs; n <= Math.min(pairs.length, policy.max_pairs); n += policy.pair_step) { + const verdict = lookVerdict(pairs.slice(0, n), policy); + if (verdict.status !== 'continue') { + if (n !== pairs.length) throw new Error(`Activity sampling continued after the verdict at ${n} pairs.`); + return verdict; + } + } + throw new Error(`Activity sampling stopped at ${pairs.length} pairs without a verdict.`); +} +export function verdictRule(summary, policy = verdictPolicy) { + const [low, high] = summary.interval_percent; + const confidence = Math.round((1 - 2 * policy.tail_alpha) * 100); + const head = `median ${summary.median_decode_degradation_percent.toFixed(2)}%, ${confidence}% interval [${low.toFixed(2)}%, ${high.toFixed(2)}%] over ${summary.paired_runs} pairs`; + if (summary.status === 'investigate') return `${head}; slowdown: the interval lies wholly above the ${policy.budget_percent}% budget`; + if (summary.status === 'within-target') return `${head}; within target: the interval reaches no further than ${policy.precision_ceiling_percent}% and does not lie above the ${policy.budget_percent}% budget`; + return `${head}; measurement noise, not a slowdown: after the ${policy.max_pairs}-pair cap the interval still straddles the ${policy.budget_percent}% budget and reaches past ${policy.precision_ceiling_percent}%`; +} +export function modeSummary(mode, pairs, policy = verdictPolicy) { + const verdict = sequentialVerdict(pairs, policy); + const degradations = pairs.map(pairDegradation); + const baseline = pairs.map((pair) => pair.off); + const mean = baseline.reduce((a, b) => a + b, 0) / baseline.length; + const cv = Math.sqrt(baseline.reduce((sum, value) => sum + (value - mean) ** 2, 0) / baseline.length) / mean * 100; + const summary = { mode, status: verdict.status, paired_runs: verdict.paired_runs, median_decode_degradation_percent: verdict.median_decode_degradation_percent, interval_percent: verdict.interval_percent, baseline_cv_percent: cv, paired_range_percent: [Math.min(...degradations), Math.max(...degradations)] }; + return { ...summary, rule: verdictRule(summary, policy) }; +} +// Pairs a recorded sample list (warmup first, then each mode's block) by adjacency: every pair is +// one `off` and one mode sample with the same pair index, `off` first on even pairs. +export function pairsFromSamples(samples) { + const measured = samples.filter((sample) => sample.mode !== 'warmup'); + if (measured.length % 2 !== 0) throw new Error('Activity samples do not form whole pairs.'); + const byMode = new Map(); + for (let i = 0; i < measured.length; i += 2) { + const [first, second] = [measured[i], measured[i + 1]]; + const off = first.mode === 'off' ? first : second; + const on = first.mode === 'off' ? second : first; + if (off.mode !== 'off' || on.mode === 'off' || first.pair !== second.pair) throw new Error(`Activity samples ${i + 1} and ${i + 2} are not one off/on pair.`); + const pairs = byMode.get(on.mode) ?? []; + if (on.pair !== pairs.length || (on.pair % 2 === 0) !== (first.mode === 'off')) throw new Error(`Activity ${on.mode} pair ${on.pair} is out of order.`); + pairs.push({ off: off.tokens_per_second, on: on.tokens_per_second }); + byMode.set(on.mode, pairs); + } + return byMode; +} diff --git a/webui/scripts/activity-performance-fixtures.json b/webui/scripts/activity-performance-fixtures.json new file mode 100644 index 000000000..a83318bb2 --- /dev/null +++ b/webui/scripts/activity-performance-fixtures.json @@ -0,0 +1,3278 @@ +{ + "description": "Shared verdict fixture for the Activity decode-overhead gate (#1925). activity-stream.spec.mjs and scripts/webui/activity_performance_verdict_tests.py both evaluate every case and must reproduce `expected` exactly. Recorded cases copy activity-performance.log samples from the webui-activity-performance-diagnostics-Linux artifacts named in `source` (copied here because artifacts expire); `look` evaluates the recorded pairs at their own count, `sequence` replays every look and requires the run to have stopped at its first decision, `pairs` checks pairing only.", + "policy": { + "budget_percent": 2, + "precision_ceiling_percent": 4, + "tail_alpha": 0.01, + "min_pairs": 12, + "pair_step": 2, + "max_pairs": 40 + }, + "cases": [ + { + "name": "recorded run 35423171315 attempt 1 PR #1920 (catalog display-name strings only); failed two-visible on CV", + "source": "run 35423171315 attempt 1, PR #1920 (catalog display-name strings only); failed two-visible on CV; artifact 10578158163, activity-performance.log as recorded", + "check": "look", + "samples": [ + { + "mode": "warmup", + "pair": -1, + "tokens_per_second": 150.44247787610618 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 146.55172413793105 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 154.54545454545456 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 153.9855072463768 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 150.97690941385434 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 155.96330275229357 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 160.075329566855 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 156.15431720759338 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 153.06122448979593 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 148.77479579929988 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 146.04810996563575 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 149.73576042278333 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 150.2651738361815 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 151.5151515151515 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 145.54794520547946 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 147.91183294663574 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 155.29841656516444 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 157.69944341372914 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 157.7970297029703 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 169.32270916334662 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 167.21311475409837 + } + ], + "expected": { + "one-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": -2.0208205756276647, + "interval_percent": null + }, + "two-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": -0.35356511490866305, + "interval_percent": null + } + } + }, + { + "name": "recorded run 35425297407 attempt 1 PR #1920; failed two-visible on CV", + "source": "run 35425297407 attempt 1, PR #1920; failed two-visible on CV; artifact 10579715535, activity-performance.log as recorded", + "check": "look", + "samples": [ + { + "mode": "warmup", + "pair": -1, + "tokens_per_second": 155.77275503970677 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 162.52390057361376 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 163.25224071702945 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 157.601977750309 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 155.203895313451 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 152.69461077844312 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 151.78571428571428 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 151.78571428571428 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 154.63917525773195 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 154.63917525773195 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 161.59695817490493 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 159.27545284197376 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 162.31699554423935 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 154.82695810564664 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 145.88100686498856 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 168.31683168316832 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 164.19832582099164 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 168.53932584269663 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 168.87417218543047 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 167.98418972332016 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 165.69200779727097 + } + ], + "expected": { + "one-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": -0.44814340588989104, + "interval_percent": null + }, + "two-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": 0.19828155981494064, + "interval_percent": null + } + } + }, + { + "name": "recorded run 35425297407 attempt 2 PR #1920; failed two-visible on the median", + "source": "run 35425297407 attempt 2, PR #1920; failed two-visible on the median; artifact 10578874615, activity-performance.log as recorded", + "check": "look", + "samples": [ + { + "mode": "warmup", + "pair": -1, + "tokens_per_second": 156.7301782421635 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 160.68052930056712 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 162.42038216560508 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 159.8746081504702 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 160.4782882315922 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 157.01970443349754 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 158.97755610972567 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 149.8237367802585 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 145.63106796116506 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 152.78609946075494 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 154.35835351089588 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 152.96940611877625 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 156.25 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 158.58208955223878 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 163.04347826086956 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 164.3041237113402 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 156.25 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 161.08654453569173 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 168.09492419248517 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 156.7301782421635 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 157.31030228254164 + } + ], + "expected": { + "one-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": -1.0828025477706835, + "interval_percent": null + }, + "two-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": 2.7363184079602108, + "interval_percent": null + } + } + }, + { + "name": "recorded run 35425297407 attempt 3 PR #1920; failed one-visible on the median", + "source": "run 35425297407 attempt 3, PR #1920; failed one-visible on the median; artifact 10580160266, activity-performance.log as recorded", + "check": "look", + "samples": [ + { + "mode": "warmup", + "pair": -1, + "tokens_per_second": 153.15315315315314 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 152.23880597014926 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 153.15315315315314 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 145.21640091116174 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 154.9210206561361 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 156.92307692307693 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 155.58267236119585 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 165.7997399219766 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 171.60161507402424 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 159.57446808510636 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 154.17170495767837 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 162.93929712460064 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 160.4782882315922 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 166.015625 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 163.46153846153845 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 160.37735849056602 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 160.37735849056602 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 163.14779270633397 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 162.93929712460064 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 156.82656826568268 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 162.00762388818296 + } + ], + "expected": { + "one-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": 3.381014304291293, + "interval_percent": null + }, + "two-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": -0.12795905310300507, + "interval_percent": null + } + } + }, + { + "name": "recorded run 35425050365 attempt 1 main after #1922; failed two-visible on CV while 6.76% faster", + "source": "run 35425050365 attempt 1, main after #1922; failed two-visible on CV while 6.76% faster; artifact 10578028841, activity-performance.log as recorded", + "check": "look", + "samples": [ + { + "mode": "warmup", + "pair": -1, + "tokens_per_second": 152.23880597014926 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 160.37735849056602 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 161.39240506329114 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 167.6528599605523 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 162.00762388818296 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 165.36964980544747 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 162.00762388818296 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 157.69944341372914 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 154.0785498489426 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 166.23207301173403 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 172.06477732793522 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 171.3709677419355 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 158.48353014294594 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 159.375 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 148.25581395348837 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 145.54794520547946 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 155.3930530164534 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 148.77479579929988 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 141.43094841930116 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 141.66666666666666 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 160.98484848484847 + } + ], + "expected": { + "one-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": -2.3500309214595005, + "interval_percent": null + }, + "two-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": -6.764168190127968, + "interval_percent": null + } + } + }, + { + "name": "recorded run 35421166203 attempt 1 PR #1919; failed two-visible on the median", + "source": "run 35421166203 attempt 1, PR #1919; failed two-visible on the median; artifact 10577094451, activity-performance.log as recorded", + "check": "look", + "samples": [ + { + "mode": "warmup", + "pair": -1, + "tokens_per_second": 159.77443609022555 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 161.08654453569173 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 166.34050880626225 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 163.67137355584083 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 172.06477732793522 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 172.99864314789687 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 168.42800528401585 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 160.17587939698493 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 160.27655562539283 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 159.67438948027552 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 161.9047619047619 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 160.37735849056602 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 154.54545454545456 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 150.70921985815605 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 168.53932584269663 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 169.21035169210353 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 164.9417852522639 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 161.59695817490493 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 156.44171779141107 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 152.78609946075494 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 147.91183294663574 + } + ], + "expected": { + "one-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": 0.06281407035175621, + "interval_percent": null + }, + "two-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": 3.1902552204176247, + "interval_percent": null + } + } + }, + { + "name": "recorded run 35567777700 attempt 1 PR #1944 under a concurrent compile; failed one-visible on the median", + "source": "run 35567777700 attempt 1, PR #1944 under a concurrent compile; failed one-visible on the median; artifact 10624751958, activity-performance.log as recorded", + "check": "look", + "samples": [ + { + "mode": "warmup", + "pair": -1, + "tokens_per_second": 162.21374045801525 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 170.34068136272543 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 162.21374045801525 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 163.566388710712 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 167.1035386631717 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 163.14779270633397 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 158.58208955223878 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 163.77649325626206 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 168.98608349900599 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 170.11340893929284 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 172.06477732793522 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 169.21035169210353 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 171.83288409703505 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 172.41379310344826 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 172.76422764227644 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 159.67438948027552 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 159.77443609022555 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 165.58441558441558 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 166.9941060903733 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 166.34050880626225 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 163.67137355584083 + } + ], + "expected": { + "one-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": 2.7985074626865805, + "interval_percent": null + }, + "two-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": 0.20283975659231238, + "interval_percent": null + } + } + }, + { + "name": "recorded run 35588958676 attempt 1 PR #1946 under a concurrent compile; failed one-visible on the median", + "source": "run 35588958676 attempt 1, PR #1946 under a concurrent compile; failed one-visible on the median; artifact 10633144458, activity-performance.log as recorded", + "check": "look", + "samples": [ + { + "mode": "warmup", + "pair": -1, + "tokens_per_second": 155.67765567765568 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 171.3709677419355 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 164.09266409266408 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 164.51612903225805 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 171.14093959731542 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 159.57446808510636 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 161.39240506329114 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 149.2100643651258 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 161.39240506329114 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 160.37735849056602 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 159.47467166979362 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 166.9941060903733 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 164.51612903225805 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 164.51612903225805 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 166.55780535597648 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 170.7970529135968 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 167.21311475409837 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 166.12377850162866 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 162.8352490421456 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 161.80203045685278 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 166.7756703727927 + } + ], + "expected": { + "one-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": 3.8709677419354853, + "interval_percent": null + }, + "two-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": 1.2258064516129108, + "interval_percent": null + } + } + }, + { + "name": "recorded run 38265665905 attempt 1 PR #2279 (YAML parser only); failed two-visible on CV", + "source": "run 38265665905 attempt 1, PR #2279 (YAML parser only); failed two-visible on CV, every pair faster; artifact 11755076778, activity-performance.log as recorded", + "check": "look", + "samples": [ + { + "mode": "warmup", + "pair": -1, + "tokens_per_second": 160.4782882315922 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 165.15544041450778 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 164.83516483516485 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 168.65079365079364 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 165.69200779727097 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 163.88174807197942 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 163.9871382636656 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 169.54787234042553 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 157.01970443349754 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 165.4769630110318 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 141.43094841930116 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 143.66197183098592 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 162.00762388818296 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 163.67137355584083 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 159.77443609022555 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 162.73133375877472 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 169.09814323607426 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 170.5685618729097 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 168.31683168316832 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 159.8746081504702 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 163.9871382636656 + } + ], + "expected": { + "one-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": -0.06430868167203967, + "interval_percent": null + }, + "two-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": -2.5723472668810388, + "interval_percent": null + } + } + }, + { + "name": "recorded run 38265665905 attempt 2 PR #2279; failed one-visible on the median", + "source": "run 38265665905 attempt 2, PR #2279; failed one-visible on the median; artifact 11754504902, activity-performance.log as recorded", + "check": "look", + "samples": [ + { + "mode": "warmup", + "pair": -1, + "tokens_per_second": 159.47467166979362 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 156.5377532228361 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 163.14779270633397 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 146.97406340057637 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 153.6144578313253 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 149.91181657848324 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 142.9372197309417 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 154.4518473652332 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 172.2972972972973 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 171.7171717171717 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 154.2649727767695 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 167.4326986211425 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 170.5685618729097 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 165.90761223162005 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 162.00762388818296 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 161.59695817490493 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 163.88174807197942 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 168.98608349900599 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 165.69200779727097 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 164.09266409266408 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 166.4490861618799 + } + ], + "expected": { + "one-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": 4.652466367713, + "interval_percent": null + }, + "two-visible": { + "status": "continue", + "paired_runs": 5, + "median_decode_degradation_percent": -1.872909698996648, + "interval_percent": null + } + } + }, + { + "name": "null: every recorded one-visible pair of the ten failed runs, in run order", + "source": "one-visible pairs of artifacts 10578158163, 10579715535, 10578874615, 10580160266, 10578028841, 10577094451, 10624751958, 10633144458, 11755076778, 11754504902 concatenated and re-sequenced into harness order; rates unchanged; cut at the first decision", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 146.55172413793105 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 154.54545454545456 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 153.9855072463768 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 150.97690941385434 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 155.96330275229357 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 160.075329566855 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 156.15431720759338 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 153.06122448979593 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 148.77479579929988 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 146.04810996563575 + }, + { + "mode": "one-visible", + "pair": 5, + "tokens_per_second": 163.25224071702945 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 162.52390057361376 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 155.203895313451 + }, + { + "mode": "one-visible", + "pair": 6, + "tokens_per_second": 157.601977750309 + }, + { + "mode": "one-visible", + "pair": 7, + "tokens_per_second": 151.78571428571428 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 152.69461077844312 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 154.63917525773195 + }, + { + "mode": "one-visible", + "pair": 8, + "tokens_per_second": 151.78571428571428 + }, + { + "mode": "one-visible", + "pair": 9, + "tokens_per_second": 161.59695817490493 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 154.63917525773195 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 160.68052930056712 + }, + { + "mode": "one-visible", + "pair": 10, + "tokens_per_second": 162.42038216560508 + }, + { + "mode": "one-visible", + "pair": 11, + "tokens_per_second": 159.8746081504702 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 160.4782882315922 + } + ], + "expected": { + "one-visible": { + "status": "within-target", + "paired_runs": 12, + "median_decode_degradation_percent": -1.3139599883476363, + "interval_percent": [ + -4.4993662864385255, + 1.832760595647187 + ] + } + } + }, + { + "name": "null: every recorded two-visible pair of the ten failed runs, in run order", + "source": "two-visible pairs of artifacts 10578158163, 10579715535, 10578874615, 10580160266, 10578028841, 10577094451, 10624751958, 10633144458, 11755076778, 11754504902 concatenated and re-sequenced into harness order; rates unchanged; cut at the first decision", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 149.73576042278333 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 150.2651738361815 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 151.5151515151515 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 145.54794520547946 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 147.91183294663574 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 155.29841656516444 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 157.69944341372914 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 157.7970297029703 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 169.32270916334662 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 167.21311475409837 + }, + { + "mode": "two-visible", + "pair": 5, + "tokens_per_second": 162.31699554423935 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 159.27545284197376 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 145.88100686498856 + }, + { + "mode": "two-visible", + "pair": 6, + "tokens_per_second": 154.82695810564664 + }, + { + "mode": "two-visible", + "pair": 7, + "tokens_per_second": 164.19832582099164 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 168.31683168316832 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 168.87417218543047 + }, + { + "mode": "two-visible", + "pair": 8, + "tokens_per_second": 168.53932584269663 + }, + { + "mode": "two-visible", + "pair": 9, + "tokens_per_second": 165.69200779727097 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 167.98418972332016 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 152.96940611877625 + }, + { + "mode": "two-visible", + "pair": 10, + "tokens_per_second": 156.25 + }, + { + "mode": "two-visible", + "pair": 11, + "tokens_per_second": 158.58208955223878 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 163.04347826086956 + } + ], + "expected": { + "two-visible": { + "status": "within-target", + "paired_runs": 12, + "median_decode_degradation_percent": -0.14586109796144953, + "interval_percent": [ + -4.993909866017052, + 2.4468770122343777 + ] + } + } + }, + { + "name": "injected 6% slowdown on the recorded two-visible pairs", + "source": "the null two-visible sequence with every Activity-arm rate multiplied by 0.94; cut at the first decision", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 149.73576042278333 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 141.2492634060106 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 142.4242424242424 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 145.54794520547946 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 147.91183294663574 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 145.98051157125457 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 148.2374768089054 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 157.7970297029703 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 169.32270916334662 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 157.18032786885246 + }, + { + "mode": "two-visible", + "pair": 5, + "tokens_per_second": 152.577975811585 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 159.27545284197376 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 145.88100686498856 + }, + { + "mode": "two-visible", + "pair": 6, + "tokens_per_second": 145.53734061930783 + }, + { + "mode": "two-visible", + "pair": 7, + "tokens_per_second": 154.34642627173213 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 168.31683168316832 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 168.87417218543047 + }, + { + "mode": "two-visible", + "pair": 8, + "tokens_per_second": 158.4269662921348 + }, + { + "mode": "two-visible", + "pair": 9, + "tokens_per_second": 155.75048732943472 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 167.98418972332016 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 152.96940611877625 + }, + { + "mode": "two-visible", + "pair": 10, + "tokens_per_second": 146.875 + }, + { + "mode": "two-visible", + "pair": 11, + "tokens_per_second": 149.06716417910445 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 163.04347826086956 + }, + { + "mode": "off", + "pair": 12, + "tokens_per_second": 164.3041237113402 + }, + { + "mode": "two-visible", + "pair": 12, + "tokens_per_second": 146.875 + }, + { + "mode": "two-visible", + "pair": 13, + "tokens_per_second": 151.42135186355023 + }, + { + "mode": "off", + "pair": 13, + "tokens_per_second": 168.09492419248517 + } + ], + "expected": { + "two-visible": { + "status": "investigate", + "paired_runs": 14, + "median_decode_degradation_percent": 6.122258505036336, + "interval_percent": [ + 2.1461675579322836, + 8.5721393034826 + ] + } + } + }, + { + "name": "injected 6% slowdown on the recorded one-visible pairs", + "source": "the null one-visible sequence with every Activity-arm rate multiplied by 0.94; cut at the first decision", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 146.55172413793105 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 145.27272727272728 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 144.7463768115942 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 150.97690941385434 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 155.96330275229357 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 150.47080979284368 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 146.78505817513778 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 153.06122448979593 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 148.77479579929988 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 137.2852233676976 + }, + { + "mode": "one-visible", + "pair": 5, + "tokens_per_second": 153.45710627400769 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 162.52390057361376 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 155.203895313451 + }, + { + "mode": "one-visible", + "pair": 6, + "tokens_per_second": 148.14585908529045 + }, + { + "mode": "one-visible", + "pair": 7, + "tokens_per_second": 142.67857142857142 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 152.69461077844312 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 154.63917525773195 + }, + { + "mode": "one-visible", + "pair": 8, + "tokens_per_second": 142.67857142857142 + }, + { + "mode": "one-visible", + "pair": 9, + "tokens_per_second": 151.90114068441062 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 154.63917525773195 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 160.68052930056712 + }, + { + "mode": "one-visible", + "pair": 10, + "tokens_per_second": 152.67515923566876 + }, + { + "mode": "one-visible", + "pair": 11, + "tokens_per_second": 150.28213166144198 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 160.4782882315922 + }, + { + "mode": "off", + "pair": 12, + "tokens_per_second": 157.01970443349754 + }, + { + "mode": "one-visible", + "pair": 12, + "tokens_per_second": 149.4389027431421 + }, + { + "mode": "one-visible", + "pair": 13, + "tokens_per_second": 140.834312573443 + }, + { + "mode": "off", + "pair": 13, + "tokens_per_second": 145.63106796116506 + } + ], + "expected": { + "one-visible": { + "status": "investigate", + "paired_runs": 14, + "median_decode_degradation_percent": 4.687759895687254, + "interval_percent": [ + 3.2937720329024893, + 6.559523809523822 + ] + } + } + }, + { + "name": "Activity arm 3% faster on the recorded one-visible pairs", + "source": "the null one-visible sequence with every Activity-arm rate multiplied by 1.03; a negative degradation never fails", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 146.55172413793105 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 159.18181818181822 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 158.60507246376812 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 150.97690941385434 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 155.96330275229357 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 164.87758945386065 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 160.83894672382118 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 153.06122448979593 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 148.77479579929988 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 150.42955326460483 + }, + { + "mode": "one-visible", + "pair": 5, + "tokens_per_second": 168.14980793854033 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 162.52390057361376 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 155.203895313451 + }, + { + "mode": "one-visible", + "pair": 6, + "tokens_per_second": 162.33003708281828 + }, + { + "mode": "one-visible", + "pair": 7, + "tokens_per_second": 156.33928571428572 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 152.69461077844312 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 154.63917525773195 + }, + { + "mode": "one-visible", + "pair": 8, + "tokens_per_second": 156.33928571428572 + }, + { + "mode": "one-visible", + "pair": 9, + "tokens_per_second": 166.44486692015207 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 154.63917525773195 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 160.68052930056712 + }, + { + "mode": "one-visible", + "pair": 10, + "tokens_per_second": 167.29299363057322 + }, + { + "mode": "one-visible", + "pair": 11, + "tokens_per_second": 164.67084639498432 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 160.4782882315922 + } + ], + "expected": { + "one-visible": { + "status": "within-target", + "paired_runs": 12, + "median_decode_degradation_percent": -4.353378787998064, + "interval_percent": [ + -7.6343472750316765, + -1.1122565864834082 + ] + } + } + }, + { + "name": "baseline drift with CV above 5% does not fail by itself", + "source": "off rates drifting linearly from 171 to 120 tok/s over twelve pairs (the drift seen in run 35425050365 attempt 1), then holding, with the recorded two-visible paired ratios of artifacts 10578028841, 10577094451, 11754504902, 10578158163, 10579715535, 10578874615, 10580160266 and 10624751958 in that order", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 171.0 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 158.14045991298943 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 178.8409090909091 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 166.36363636363637 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 161.72727272727272 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 172.6667774638524 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 165.24790495385594 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 157.0909090909091 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 152.45454545454544 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 173.24380165289253 + }, + { + "mode": "two-visible", + "pair": 5, + "tokens_per_second": 142.4429752066116 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 147.8181818181818 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 143.1818181818182 + }, + { + "mode": "two-visible", + "pair": 6, + "tokens_per_second": 128.03433268858802 + }, + { + "mode": "two-visible", + "pair": 7, + "tokens_per_second": 135.0504527813713 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 138.54545454545456 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 133.9090909090909 + }, + { + "mode": "two-visible", + "pair": 8, + "tokens_per_second": 138.3218112685793 + }, + { + "mode": "two-visible", + "pair": 9, + "tokens_per_second": 125.14859734233283 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 129.27272727272725 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 124.63636363636363 + }, + { + "mode": "two-visible", + "pair": 10, + "tokens_per_second": 126.97069017938581 + }, + { + "mode": "two-visible", + "pair": 11, + "tokens_per_second": 122.88874430709174 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 120.0 + }, + { + "mode": "off", + "pair": 12, + "tokens_per_second": 120.0 + }, + { + "mode": "two-visible", + "pair": 12, + "tokens_per_second": 121.69665809768637 + }, + { + "mode": "two-visible", + "pair": 13, + "tokens_per_second": 122.38568588469185 + }, + { + "mode": "off", + "pair": 13, + "tokens_per_second": 120.0 + } + ], + "expected": { + "two-visible": { + "status": "within-target", + "paired_runs": 14, + "median_decode_degradation_percent": -1.9304906347865962, + "interval_percent": [ + -6.764168190127973, + 3.6363636363636083 + ], + "baseline_cv_percent": 12.193276713672626 + } + } + }, + { + "name": "noise too wide to decide ends inconclusive at the cap", + "source": "the first 40 null two-visible pairs with each degradation moved to 2.5 + 3 x (its distance from the median); never decided before the cap", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 149.73576042278333 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 147.71950809992475 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 159.9458821511169 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 145.54794520547946 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 147.91183294663574 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 166.5109974710853 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 153.7057245265343 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 157.7970297029703 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 169.32270916334662 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 158.9179293652968 + }, + { + "mode": "two-visible", + "pair": 5, + "tokens_per_second": 164.56594551161908 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 159.27545284197376 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 145.88100686498856 + }, + { + "mode": "two-visible", + "pair": 6, + "tokens_per_second": 169.20716102467478 + }, + { + "mode": "two-visible", + "pair": 7, + "tokens_per_second": 151.90953136734493 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 168.31683168316832 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 168.87417218543047 + }, + { + "mode": "two-visible", + "pair": 8, + "tokens_per_second": 163.80443392883194 + }, + { + "mode": "two-visible", + "pair": 9, + "tokens_per_second": 157.06386869162884 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 167.98418972332016 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 152.96940611877625 + }, + { + "mode": "two-visible", + "pair": 10, + "tokens_per_second": 159.12885372832667 + }, + { + "mode": "two-visible", + "pair": 11, + "tokens_per_second": 145.73447144771416 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 163.04347826086956 + }, + { + "mode": "off", + "pair": 12, + "tokens_per_second": 164.3041237113402 + }, + { + "mode": "two-visible", + "pair": 12, + "tokens_per_second": 136.1865651837117 + }, + { + "mode": "two-visible", + "pair": 13, + "tokens_per_second": 143.02334432897405 + }, + { + "mode": "off", + "pair": 13, + "tokens_per_second": 168.09492419248517 + }, + { + "mode": "off", + "pair": 14, + "tokens_per_second": 156.7301782421635 + }, + { + "mode": "two-visible", + "pair": 14, + "tokens_per_second": 154.69768568297863 + }, + { + "mode": "two-visible", + "pair": 15, + "tokens_per_second": 151.63393764373538 + }, + { + "mode": "off", + "pair": 15, + "tokens_per_second": 162.93929712460064 + }, + { + "mode": "off", + "pair": 16, + "tokens_per_second": 163.46153846153845 + }, + { + "mode": "two-visible", + "pair": 16, + "tokens_per_second": 167.1888936955901 + }, + { + "mode": "two-visible", + "pair": 17, + "tokens_per_second": 156.5166975881261 + }, + { + "mode": "off", + "pair": 17, + "tokens_per_second": 160.37735849056602 + }, + { + "mode": "off", + "pair": 18, + "tokens_per_second": 162.93929712460064 + }, + { + "mode": "two-visible", + "pair": 18, + "tokens_per_second": 159.64245106796074 + }, + { + "mode": "two-visible", + "pair": 19, + "tokens_per_second": 168.5945501178825 + }, + { + "mode": "off", + "pair": 19, + "tokens_per_second": 156.82656826568268 + }, + { + "mode": "off", + "pair": 20, + "tokens_per_second": 171.3709677419355 + }, + { + "mode": "two-visible", + "pair": 20, + "tokens_per_second": 128.58335196453712 + }, + { + "mode": "two-visible", + "pair": 21, + "tokens_per_second": 178.04450532855844 + }, + { + "mode": "off", + "pair": 21, + "tokens_per_second": 148.25581395348837 + }, + { + "mode": "off", + "pair": 22, + "tokens_per_second": 145.54794520547946 + }, + { + "mode": "two-visible", + "pair": 22, + "tokens_per_second": 171.57958665502258 + }, + { + "mode": "two-visible", + "pair": 23, + "tokens_per_second": 160.05791438909242 + }, + { + "mode": "off", + "pair": 23, + "tokens_per_second": 141.43094841930116 + }, + { + "mode": "off", + "pair": 24, + "tokens_per_second": 141.66666666666666 + }, + { + "mode": "two-visible", + "pair": 24, + "tokens_per_second": 196.2109616573902 + }, + { + "mode": "two-visible", + "pair": 25, + "tokens_per_second": 139.02098575279172 + }, + { + "mode": "off", + "pair": 25, + "tokens_per_second": 160.37735849056602 + }, + { + "mode": "off", + "pair": 26, + "tokens_per_second": 168.53932584269663 + }, + { + "mode": "two-visible", + "pair": 26, + "tokens_per_second": 110.99186920111755 + }, + { + "mode": "two-visible", + "pair": 27, + "tokens_per_second": 152.3313605113508 + }, + { + "mode": "off", + "pair": 27, + "tokens_per_second": 169.21035169210353 + }, + { + "mode": "off", + "pair": 28, + "tokens_per_second": 156.44171779141107 + }, + { + "mode": "two-visible", + "pair": 28, + "tokens_per_second": 168.1415181842979 + }, + { + "mode": "two-visible", + "pair": 29, + "tokens_per_second": 134.485378507445 + }, + { + "mode": "off", + "pair": 29, + "tokens_per_second": 152.78609946075494 + }, + { + "mode": "off", + "pair": 30, + "tokens_per_second": 169.21035169210353 + }, + { + "mode": "two-visible", + "pair": 30, + "tokens_per_second": 173.0046570456642 + }, + { + "mode": "two-visible", + "pair": 31, + "tokens_per_second": 167.55408199674076 + }, + { + "mode": "off", + "pair": 31, + "tokens_per_second": 172.76422764227644 + }, + { + "mode": "off", + "pair": 32, + "tokens_per_second": 159.67438948027552 + }, + { + "mode": "two-visible", + "pair": 32, + "tokens_per_second": 156.13079052810966 + }, + { + "mode": "two-visible", + "pair": 33, + "tokens_per_second": 158.74509296485152 + }, + { + "mode": "off", + "pair": 33, + "tokens_per_second": 166.9941060903733 + }, + { + "mode": "off", + "pair": 34, + "tokens_per_second": 166.34050880626225 + }, + { + "mode": "two-visible", + "pair": 34, + "tokens_per_second": 154.32889507399378 + }, + { + "mode": "two-visible", + "pair": 35, + "tokens_per_second": 155.54023330837893 + }, + { + "mode": "off", + "pair": 35, + "tokens_per_second": 166.9941060903733 + }, + { + "mode": "off", + "pair": 36, + "tokens_per_second": 166.55780535597648 + }, + { + "mode": "two-visible", + "pair": 36, + "tokens_per_second": 156.42333756386793 + }, + { + "mode": "two-visible", + "pair": 37, + "tokens_per_second": 155.93375093685677 + }, + { + "mode": "off", + "pair": 37, + "tokens_per_second": 170.7970529135968 + }, + { + "mode": "off", + "pair": 38, + "tokens_per_second": 162.8352490421456 + }, + { + "mode": "two-visible", + "pair": 38, + "tokens_per_second": 168.78100930125927 + }, + { + "mode": "two-visible", + "pair": 39, + "tokens_per_second": 172.8279940911703 + }, + { + "mode": "off", + "pair": 39, + "tokens_per_second": 161.80203045685278 + } + ], + "expected": { + "two-visible": { + "status": "inconclusive", + "paired_runs": 40, + "median_decode_degradation_percent": 2.5000000000000027, + "interval_percent": [ + -3.321599515334178, + 6.858848524747212 + ] + } + } + }, + { + "name": "an interval touching the budget is not wholly above it", + "source": "synthetic: 12 pairs at exactly 2% degradation", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 100.0 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 98.0 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 98.0 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 100.0 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 100.0 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 98.0 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 98.0 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 100.0 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 100.0 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 98.0 + }, + { + "mode": "one-visible", + "pair": 5, + "tokens_per_second": 98.0 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 100.0 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 100.0 + }, + { + "mode": "one-visible", + "pair": 6, + "tokens_per_second": 98.0 + }, + { + "mode": "one-visible", + "pair": 7, + "tokens_per_second": 98.0 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 100.0 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 100.0 + }, + { + "mode": "one-visible", + "pair": 8, + "tokens_per_second": 98.0 + }, + { + "mode": "one-visible", + "pair": 9, + "tokens_per_second": 98.0 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 100.0 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 100.0 + }, + { + "mode": "one-visible", + "pair": 10, + "tokens_per_second": 98.0 + }, + { + "mode": "one-visible", + "pair": 11, + "tokens_per_second": 98.0 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 100.0 + } + ], + "expected": { + "one-visible": { + "status": "within-target", + "paired_runs": 12, + "median_decode_degradation_percent": 2.0, + "interval_percent": [ + 2.0, + 2.0 + ] + } + } + }, + { + "name": "sampling that continued past its verdict is rejected", + "source": "the null one-visible sequence with two pairs more than its decision", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 146.55172413793105 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 154.54545454545456 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 153.9855072463768 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 150.97690941385434 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 155.96330275229357 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 160.075329566855 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 156.15431720759338 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 153.06122448979593 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 148.77479579929988 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 146.04810996563575 + }, + { + "mode": "one-visible", + "pair": 5, + "tokens_per_second": 163.25224071702945 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 162.52390057361376 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 155.203895313451 + }, + { + "mode": "one-visible", + "pair": 6, + "tokens_per_second": 157.601977750309 + }, + { + "mode": "one-visible", + "pair": 7, + "tokens_per_second": 151.78571428571428 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 152.69461077844312 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 154.63917525773195 + }, + { + "mode": "one-visible", + "pair": 8, + "tokens_per_second": 151.78571428571428 + }, + { + "mode": "one-visible", + "pair": 9, + "tokens_per_second": 161.59695817490493 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 154.63917525773195 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 160.68052930056712 + }, + { + "mode": "one-visible", + "pair": 10, + "tokens_per_second": 162.42038216560508 + }, + { + "mode": "one-visible", + "pair": 11, + "tokens_per_second": 159.8746081504702 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 160.4782882315922 + }, + { + "mode": "off", + "pair": 12, + "tokens_per_second": 157.01970443349754 + }, + { + "mode": "one-visible", + "pair": 12, + "tokens_per_second": 158.97755610972567 + }, + { + "mode": "one-visible", + "pair": 13, + "tokens_per_second": 149.8237367802585 + }, + { + "mode": "off", + "pair": 13, + "tokens_per_second": 145.63106796116506 + } + ], + "expected": { + "error": "continued after the verdict" + } + }, + { + "name": "sampling that stopped before a verdict is rejected", + "source": "the inconclusive sequence cut at 13 pairs", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 149.73576042278333 + }, + { + "mode": "two-visible", + "pair": 0, + "tokens_per_second": 147.71950809992475 + }, + { + "mode": "two-visible", + "pair": 1, + "tokens_per_second": 159.9458821511169 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 145.54794520547946 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 147.91183294663574 + }, + { + "mode": "two-visible", + "pair": 2, + "tokens_per_second": 166.5109974710853 + }, + { + "mode": "two-visible", + "pair": 3, + "tokens_per_second": 153.7057245265343 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 157.7970297029703 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 169.32270916334662 + }, + { + "mode": "two-visible", + "pair": 4, + "tokens_per_second": 158.9179293652968 + }, + { + "mode": "two-visible", + "pair": 5, + "tokens_per_second": 164.56594551161908 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 159.27545284197376 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 145.88100686498856 + }, + { + "mode": "two-visible", + "pair": 6, + "tokens_per_second": 169.20716102467478 + }, + { + "mode": "two-visible", + "pair": 7, + "tokens_per_second": 151.90953136734493 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 168.31683168316832 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 168.87417218543047 + }, + { + "mode": "two-visible", + "pair": 8, + "tokens_per_second": 163.80443392883194 + }, + { + "mode": "two-visible", + "pair": 9, + "tokens_per_second": 157.06386869162884 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 167.98418972332016 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 152.96940611877625 + }, + { + "mode": "two-visible", + "pair": 10, + "tokens_per_second": 159.12885372832667 + }, + { + "mode": "two-visible", + "pair": 11, + "tokens_per_second": 145.73447144771416 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 163.04347826086956 + }, + { + "mode": "off", + "pair": 12, + "tokens_per_second": 164.3041237113402 + }, + { + "mode": "two-visible", + "pair": 12, + "tokens_per_second": 136.1865651837117 + }, + { + "mode": "two-visible", + "pair": 13, + "tokens_per_second": 143.02334432897405 + }, + { + "mode": "off", + "pair": 13, + "tokens_per_second": 168.09492419248517 + } + ], + "expected": { + "error": "without a verdict" + } + }, + { + "name": "a zero rate is rejected", + "source": "synthetic", + "check": "sequence", + "samples": [ + { + "mode": "off", + "pair": 0, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 0 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 150.0 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 149.0 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 150.0 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 149.0 + }, + { + "mode": "one-visible", + "pair": 5, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 150.0 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 6, + "tokens_per_second": 149.0 + }, + { + "mode": "one-visible", + "pair": 7, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 150.0 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 8, + "tokens_per_second": 149.0 + }, + { + "mode": "one-visible", + "pair": 9, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 150.0 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 10, + "tokens_per_second": 149.0 + }, + { + "mode": "one-visible", + "pair": 11, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 150.0 + } + ], + "expected": { + "error": "finite and positive" + } + }, + { + "name": "a pair recorded in the wrong order is rejected", + "source": "synthetic: pair 0 with the Activity arm first", + "check": "pairs", + "samples": [ + { + "mode": "one-visible", + "pair": 0, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 0, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 1, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 1, + "tokens_per_second": 150.0 + }, + { + "mode": "off", + "pair": 2, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 2, + "tokens_per_second": 149.0 + }, + { + "mode": "one-visible", + "pair": 3, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 3, + "tokens_per_second": 150.0 + }, + { + "mode": "off", + "pair": 4, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 4, + "tokens_per_second": 149.0 + }, + { + "mode": "one-visible", + "pair": 5, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 5, + "tokens_per_second": 150.0 + }, + { + "mode": "off", + "pair": 6, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 6, + "tokens_per_second": 149.0 + }, + { + "mode": "one-visible", + "pair": 7, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 7, + "tokens_per_second": 150.0 + }, + { + "mode": "off", + "pair": 8, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 8, + "tokens_per_second": 149.0 + }, + { + "mode": "one-visible", + "pair": 9, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 9, + "tokens_per_second": 150.0 + }, + { + "mode": "off", + "pair": 10, + "tokens_per_second": 150.0 + }, + { + "mode": "one-visible", + "pair": 10, + "tokens_per_second": 149.0 + }, + { + "mode": "one-visible", + "pair": 11, + "tokens_per_second": 149.0 + }, + { + "mode": "off", + "pair": 11, + "tokens_per_second": 150.0 + } + ], + "expected": { + "error": "out of order" + } + } + ] +} diff --git a/webui/scripts/activity-performance-policy.json b/webui/scripts/activity-performance-policy.json new file mode 100644 index 000000000..18c212397 --- /dev/null +++ b/webui/scripts/activity-performance-policy.json @@ -0,0 +1,8 @@ +{ + "budget_percent": 2, + "precision_ceiling_percent": 4, + "tail_alpha": 0.01, + "min_pairs": 12, + "pair_step": 2, + "max_pairs": 40 +} diff --git a/webui/scripts/activity-performance-verdict.spec.mjs b/webui/scripts/activity-performance-verdict.spec.mjs new file mode 100644 index 000000000..90e96153d --- /dev/null +++ b/webui/scripts/activity-performance-verdict.spec.mjs @@ -0,0 +1,67 @@ +// Copyright 2026 Lablup Inc. Licensed under the Apache License, Version 2.0. +/* global URL */ +// The Activity decode-overhead verdict (#1925), checked against the fixture that +// scripts/webui/activity_performance_verdict_tests.py evaluates too. +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { verdictPolicy, validatePolicy, signTestRank, medianInterval, lookVerdict, modeSummary, pairsFromSamples, performanceMode, performanceCompletion } from './activity-performance-config.mjs'; + +const fixture = JSON.parse(readFileSync(new URL('./activity-performance-fixtures.json', import.meta.url), 'utf8')); +const pick = (value, expected) => Object.fromEntries(Object.keys(expected).map((key) => [key, value[key]])); + +test('fixture was generated under the policy in force', () => { + assert.deepEqual(fixture.policy, { ...verdictPolicy }); +}); +for (const item of fixture.cases) { + test(`fixture: ${item.name}`, () => { + const evaluate = () => { + const byMode = pairsFromSamples(item.samples); + if (item.check === 'pairs') return {}; + return Object.fromEntries([...byMode].map(([mode, pairs]) => [mode, item.check === 'look' ? lookVerdict(pairs) : modeSummary(mode, pairs)])); + }; + if (item.expected.error) { + assert.throws(evaluate, (error) => error.message.includes(item.expected.error)); + return; + } + const result = evaluate(); + assert.deepEqual(Object.keys(result).sort(), Object.keys(item.expected).sort()); + for (const [mode, expected] of Object.entries(item.expected)) assert.deepEqual(pick(result[mode], expected), expected, `${item.name}: ${mode}`); + }); +} +test('none of the recorded runs that failed on noise is a slowdown', () => { + const recorded = fixture.cases.filter((item) => item.name.startsWith('recorded ')); + assert.ok(recorded.length >= 10); + for (const item of recorded) for (const verdict of Object.values(item.expected)) assert.notEqual(verdict.status, 'investigate'); +}); +test('sign-test rank matches the exact binomial tail', () => { + // P(Bin(12, 1/2) <= 1) = 13/4096 <= 0.01 < P(<= 2) = 79/4096, so the 98% interval is [x(2), x(11)]. + assert.equal(signTestRank(12, 0.01), 2); + assert.equal(signTestRank(6, 0.01), 0); + assert.equal(signTestRank(7, 0.01), 1); + assert.equal(signTestRank(40, 0.01), 13); + assert.deepEqual(medianInterval([5, 1, 4, 2, 3, 6, 7, 8, 9, 10, 11, 12]), { median: 6.5, interval: [2, 11] }); + assert.equal(medianInterval([1, 2, 3]).interval, null); +}); +test('looks happen only at min_pairs plus whole steps', () => { + const pairs = (n, on) => Array.from({ length: n }, () => ({ off: 100, on })); + assert.equal(lookVerdict(pairs(11, 90)).status, 'continue'); + assert.equal(lookVerdict(pairs(13, 90)).status, 'continue'); + assert.equal(lookVerdict(pairs(12, 90)).status, 'investigate'); + assert.equal(lookVerdict(pairs(12, 101)).status, 'within-target'); +}); +test('policy validation rejects unsafe or incomplete policies', () => { + assert.doesNotThrow(() => validatePolicy({ ...verdictPolicy })); + for (const change of [{ tail_alpha: 0 }, { tail_alpha: 0.5 }, { min_pairs: 6 }, { pair_step: 1 }, { max_pairs: 52 }, { max_pairs: 41 }, { precision_ceiling_percent: 1 }, { budget_percent: Number.NaN }, { extra: 1 }]) { + assert.throws(() => validatePolicy({ ...verdictPolicy, ...change }), /policy/); + } + const missing = { ...verdictPolicy }; + delete missing.max_pairs; + assert.throws(() => validatePolicy(missing), /policy/); +}); +test('full mode reports an undecided mode as inconclusive, never as a slowdown', () => { + const full = performanceMode(); + assert.equal(performanceCompletion(full, [{ status: 'within-target' }, { status: 'inconclusive' }]).status, 'inconclusive'); + assert.equal(performanceCompletion(full, [{ status: 'inconclusive' }, { status: 'investigate' }]).status, 'investigate'); + assert.equal(performanceCompletion(performanceMode('visible-only-headed'), [{ status: 'inconclusive' }]).status, 'incomplete'); +}); diff --git a/webui/scripts/activity-performance.mjs b/webui/scripts/activity-performance.mjs index 432a1aa51..0323492ac 100644 --- a/webui/scripts/activity-performance.mjs +++ b/webui/scripts/activity-performance.mjs @@ -3,7 +3,7 @@ // Root-owned real acceptance. Never run alongside unrelated GPU inference/tests. import { readFile, writeFile } from 'node:fs/promises'; import { chromium } from '@playwright/test'; -import { performanceMode, assertGeometry, performanceCompletion } from './activity-performance-config.mjs'; +import { performanceMode, assertGeometry, performanceCompletion, verdictPolicy, lookVerdict, modeSummary } from './activity-performance-config.mjs'; import { capturePerformanceDiagnostics } from './activity-performance-diagnostics.mjs'; const base = new URL(process.env.WEBUI_PERF_BASE ?? 'http://127.0.0.1:8080/'); @@ -122,7 +122,6 @@ async function request(mode, pair) { await writeFile(output, JSON.stringify({ status: 'in-progress', scope: modeConfig.name, hidden_native: modeConfig.headless ? 'not-run' : 'pending', preflight, samples: results }, null, 2)); return rate; } -function median(values) { const sorted = [...values].sort((a, b) => a - b); return sorted[Math.floor(sorted.length / 2)]; } try { // Validate geometry and actual visibility for every requested condition before // even the GPU warmup. Never synthesize visibility or bypass failed clicks. @@ -130,18 +129,19 @@ try { await clients('off'); await request('warmup', -1); const summaries = []; for (const mode of modeConfig.modes) { - const paired = []; const baseline = []; - for (let pair = 0; pair < 5; pair++) { - const rates = {}; + // Alternating off/on pairs until the verdict decides (#1925): the pair count is not fixed, it + // grows by pair_step from min_pairs while the interval is too wide to decide, up to max_pairs. + const pairs = []; + for (let verdict = { status: 'continue' }; verdict.status === 'continue';) { + const pair = pairs.length; const rates = {}; for (const condition of pair % 2 === 0 ? ['off', mode] : [mode, 'off']) { await clients(condition); rates[condition] = await request(condition, pair); } - baseline.push(rates.off); paired.push((rates.off - rates[mode]) / rates.off * 100); + pairs.push({ off: rates.off, on: rates[mode] }); + verdict = lookVerdict(pairs); } - const mean = baseline.reduce((a, b) => a + b, 0) / baseline.length; - const cv = Math.sqrt(baseline.reduce((sum, value) => sum + (value - mean) ** 2, 0) / baseline.length) / mean * 100; - const degradation = median(paired); - summaries.push({ mode, paired_runs: 5, median_decode_degradation_percent: degradation, baseline_cv_percent: cv, paired_range_percent: [Math.min(...paired), Math.max(...paired)], status: degradation > 2 || cv > 5 ? 'investigate' : 'within-target' }); + const summary = modeSummary(mode, pairs); + summaries.push(summary); process.stdout.write(`${mode}: ${summary.status}: ${summary.rule}\n`); } - await writeFile(output, JSON.stringify({ ...performanceCompletion(modeConfig, summaries), preflight, source: 'native completion backend timings.predicted_per_second; no browser or word-count estimate', samples: results, summaries }, null, 2)); + await writeFile(output, JSON.stringify({ ...performanceCompletion(modeConfig, summaries), preflight, source: 'native completion backend timings.predicted_per_second; no browser or word-count estimate', verdict_policy: verdictPolicy, samples: results, summaries }, null, 2)); } catch (error) { await writeFile(output, JSON.stringify({ status: 'failed', scope: modeConfig.name, hidden_native: modeConfig.headless ? 'not-run' : 'not-completed', preflight, samples: results, error: error instanceof Error ? error.message.replaceAll(key, '[redacted]') : 'unknown' }, null, 2)); throw error; diff --git a/webui/scripts/activity-stream.spec.mjs b/webui/scripts/activity-stream.spec.mjs index 70f60c97a..ea3ea389e 100644 --- a/webui/scripts/activity-stream.spec.mjs +++ b/webui/scripts/activity-stream.spec.mjs @@ -37,7 +37,10 @@ test('rejects invalid UTF-8 with a fixed safe error', async () => { await assert.rejects(validateChatStream({ read: async () => ({ done: sent, value: sent ? undefined : (sent = true, new Uint8Array([255])) }) }), /UTF-8 decoding failed/); }); -// Shared pure policy tests run in the same offline acceptance gate. +// Shared pure policy tests run in the same offline acceptance gate. The verdict tests (#1925) are +// registered from here because package.json, which lists the unit files, is part of the WebUI +// bundle's source digest. +import './activity-performance-verdict.spec.mjs'; import { performanceMode, assertGeometry, performanceCompletion } from './activity-performance-config.mjs'; test('performance default remains headed and includes actual hidden mode', () => { assert.deepEqual(performanceMode(), { name: 'full', headless: false, modes: ['one-visible', 'two-visible', 'hidden'] }); @@ -52,7 +55,7 @@ test('headed visible-only keeps the hidden acceptance unclaimed', () => { const mode = performanceMode('visible-only-headed'); assert.equal(mode.headless, false); assert.deepEqual(mode.modes, ['one-visible', 'two-visible']); - for (const status of ['within-target', 'investigate']) { + for (const status of ['within-target', 'investigate', 'inconclusive']) { const result = performanceCompletion(mode, [{ status }]); assert.equal(result.status, 'incomplete'); assert.equal(result.hidden_native, 'not-run'); @@ -63,7 +66,7 @@ test('headed visible-only keeps the hidden acceptance unclaimed', () => { test('visible-only diagnostic never reports complete acceptance', () => { const mode = performanceMode('visible-only-headless'); assert.deepEqual(mode.modes, ['one-visible', 'two-visible']); - for (const status of ['within-target', 'investigate']) { + for (const status of ['within-target', 'investigate', 'inconclusive']) { const result = performanceCompletion(mode, [{ status }]); assert.equal(result.status, 'incomplete'); assert.equal(result.hidden_native, 'not-run');