[nicoloangileri]

Checklist before merging this PR:

• Mentioned all issues that this PR fixes or addresses.
• Summarized the updates of this PR under Summary.
• Added an entry under Unreleased in the Changelog.

There is no open issue for this; the details are below.

Summary

ConformalNaiveModel and ConformalQRModel compute the interval correction as np.quantile(scores, q=alpha, method="higher") over the n calibration scores. That is the (ceil((n - 1) * alpha) + 1)-th smallest score. Split conformal prediction takes the ceil((n + 1) * alpha)-th smallest score, which is what gives coverage of at least alpha for exchangeable scores (Vovk et al. 2005; Lei et al. 2018; Angelopoulos & Bates 2021). For alpha > 0.5 the current rank can be one lower, and then the interval under-covers.

With cal_length=12 and quantiles [0.05, 0.5, 0.95] (a 90% interval), the current rule picks the 11th of 12 scores, so the guaranteed coverage is 11/13 = 84.6%. For a 90% interval the guarantee drops below 90% for 156 of the values of n between 5 and 200.

Empirical coverage on i.i.d. N(0, 1) series, with a LinearRegressionModel(lags=1) fitted on separate data and ConformalNaiveModel(quantiles=[0.05, 0.5, 0.95], cal_length=n). The numbers are one-step historical_forecasts over 20 series of length 1200, scored with mic(q_interval=(0.05, 0.95)):

n	before	after	rank/(n+1) after
6	0.857	0.857	0.857 (rank capped at n)
12	0.847	0.923	0.923
20	0.905	0.904	0.905
50	0.902	0.900	0.902

Changes:

• New _conformal_quantile() in conformal_models.py, used by both models. It selects the ceil((n + 1) * alpha)-th smallest score. When that rank is larger than n (i.e. n < alpha / (1 - alpha)), it uses the largest score, which is what happens today. Selecting the rank goes through np.quantile(..., method="nearest") at the exact position rank - 1, so NaN handling is unchanged. I checked this against a sort-based selection for every n up to 1499 and 10 levels.
• Test helper helper_compute_pred_cal builds the expected intervals with the same rule, via an independent sort-based helper_conformal_quantile.
• test_single_ts asserts that predicting with a different n changes the result. With the 17-point series and an 80% interval, both predictions now select the largest of the few residuals per step, so that check uses a narrower interval [0.3, 0.5, 0.7]. It passes before and after the change.
• New test_finite_sample_conformal_quantile (symmetric and asymmetric): with 12 calibration residuals and an 80% interval, the half-widths are the 11th smallest absolute error (symmetric) and the 12th smallest error per tail (asymmetric). It fails on master.

test_conformal_model.py: 738 passed. ruff 0.7.2 check and format pass.

Other Information

The intervals get slightly wider only when the old rank was lower than ceil((n + 1) * alpha). For large n the difference disappears.

[nicoloangileri]

Hi @dennisbader, a gentle ping on this PR when you have a moment. Happy to adjust anything if needed. Thanks!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.14%. Comparing base (275b255) to head (1b39923).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3227   +/-   ##
=======================================
  Coverage   97.14%   97.14%           
=======================================
  Files         174      174           
  Lines       19299    19304    +5     
=======================================
+ Hits        18748    18753    +5     
  Misses        551      551           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dennisbader]

Thanks a lot for fixing this and your first contribution @nicoloangileri 🚀

Everything looks good to 💯