The module header (src/review/risk-control.ts ~6-9) states:
λ̂ is the smallest λ (largest coverage) whose bound — and every more-conservative λ's bound — stays ≤ α.
The implementation (~108-130) walks candidates ascending (permissive → conservative) and returns at the first λ whose Clopper–Pearson bound ≤ α. It never checks any more-conservative λ, and it never stops on a failure — it keeps testing.
That is not fixed-sequence testing. Fixed-sequence (Learn-then-Test, the cited method) controls FWER at δ by testing a pre-specified ordering and stopping at the first non-rejection. What is implemented is "test up to K nested hypotheses, each at level δ, publish the first that passes" — a selection with no multiplicity correction. Two further deviations: the grid is the set of observed confidences (data-dependent, not pre-registered), and K is unbounded (one candidate per distinct float).
Consequence: the published statement says "with confidence 1−δ = 95%". The actual confidence level is lower and is computed nowhere. The nested structure means the inflation is well below a naive K·δ union bound, and Clopper–Pearson is demanding at small n, so I would not claim the current live number is wrong — but the advertised δ is not the δ the procedure delivers, and this is the first thing an outside statistician will check on a publicly advertised guarantee.
Note the second doc comment (~88-91) explicitly argues for the ascending direction on power grounds — so this appears to be a deliberate trade that the header and the public claim were never updated to match.
Verified sound (so nobody re-checks): clopperPearsonUpperBound, binomialCdf, and minimumCalibrationLabels are mathematically correct. Bisection direction is right, it converges conservatively (return hi), the errors >= n and n <= 0 guards are right, the zero-error case reduces exactly to 1 − δ^(1/n), and minimumCalibrationLabels = ceil(ln δ / ln(1−α)) gives 59 at α=0.05/δ=0.05 and 598 at α=0.005 as expected. The stratum-dropping bookkeeping is also correct — n at each candidate is exactly the count with confidence ≥ λ.
Fix
Either (a) restrict the grid to a pre-registered λ set and implement true fixed-sequence (test descending, stop at first failure, output the last passing λ), or (b) keep the ascending scan and Bonferroni-split δ across the K candidates actually tested, then publish the corrected level. Update the header either way.
The module header (
src/review/risk-control.ts~6-9) states:The implementation (~108-130) walks candidates ascending (permissive → conservative) and
returns at the first λ whose Clopper–Pearson bound ≤ α. It never checks any more-conservative λ, and it never stops on a failure — it keeps testing.That is not fixed-sequence testing. Fixed-sequence (Learn-then-Test, the cited method) controls FWER at δ by testing a pre-specified ordering and stopping at the first non-rejection. What is implemented is "test up to K nested hypotheses, each at level δ, publish the first that passes" — a selection with no multiplicity correction. Two further deviations: the grid is the set of observed confidences (data-dependent, not pre-registered), and K is unbounded (one candidate per distinct float).
Consequence: the published statement says "with confidence 1−δ = 95%". The actual confidence level is lower and is computed nowhere. The nested structure means the inflation is well below a naive
K·δunion bound, and Clopper–Pearson is demanding at small n, so I would not claim the current live number is wrong — but the advertised δ is not the δ the procedure delivers, and this is the first thing an outside statistician will check on a publicly advertised guarantee.Note the second doc comment (~88-91) explicitly argues for the ascending direction on power grounds — so this appears to be a deliberate trade that the header and the public claim were never updated to match.
Verified sound (so nobody re-checks):
clopperPearsonUpperBound,binomialCdf, andminimumCalibrationLabelsare mathematically correct. Bisection direction is right, it converges conservatively (return hi), theerrors >= nandn <= 0guards are right, the zero-error case reduces exactly to1 − δ^(1/n), andminimumCalibrationLabels = ceil(ln δ / ln(1−α))gives 59 at α=0.05/δ=0.05 and 598 at α=0.005 as expected. The stratum-dropping bookkeeping is also correct —nat each candidate is exactly the count withconfidence ≥ λ.Fix
Either (a) restrict the grid to a pre-registered λ set and implement true fixed-sequence (test descending, stop at first failure, output the last passing λ), or (b) keep the ascending scan and Bonferroni-split δ across the K candidates actually tested, then publish the corrected level. Update the header either way.