[zzylol]

Closes #158. Stacked on #178.

Why

Since #171, SaturationCurves::accuracy reads every sketch's plain curve at the lookback's item count, whatever the window. A KLL or top-k answer merged from L/x windows therefore got the unmerged sketch's accuracy, although #131 measured a merge penalty: KLL error up to 1.3× at k = 800 when saturated, and top-k precision down by up to 40% at large K. AutoSketch never merges, so the AutoSketch vs. ASAP evaluation favoured ASAP.

What

• SaturationCurves::load also reads saturation_merge_curve.csv where present: shard counts above 1, per grid point. A later run's point (out_1e9) keeps an earlier run's merge curves when it has none.

• accuracy: for kll-percall and the heap top-k variants, m = L / x. It reads the merge curve at the lookback's item count (the study splits the same items into m shards):

  • between measured counts, the worse one;
  • past the largest measured count, the largest;
  • one window, the plain curve;
  • no merge curve, no accuracy.

  The other sketches merge exactly and are unchanged.

• Docs: rqe_sketch_deployment_v1.md eligibility and v1 scope.

Limits

Merge curves cover m ≤ 64 and N ≤ 1e7. Past N = 1e7 the merge curve's last value is used if the plain curve saturated (same rule as the plain curve).

Verification

• cargo test -p rqe-optimizer (61): new tests for bracketing shard counts, past the largest, no merge curve, and loading across runs.
• clippy -D warnings and fmt are clean.
• Real merge curves (topk, quantile; m = 1, 4, 16, 64) are being regenerated on clnode109 in the rqe-optimizer: one study for sketch cost and accuracy (#174) #178 rerun.

🤖 Generated with Claude Code

[zzylol]

@milindsrivastava1997 can you confirm this change to the #171 behavior?

#156 Q2 chose "B now, C as a follow-up (#158)" for the KLL/top-k merge penalty, and the #171 amendment made KLL and top-k read the curve at the lookback's item count like every other sketch, leaving the penalty to #158. When #158 was closed, the reason given was that merge_accuracy + accuracy_at already covered it. After #171, though, sketches never read the cost table's accuracy, so the penalty wasn't modeled anywhere. (#178 removes merge_accuracy.)

This PR is option C from Q2: for KLL and top-k merged from m = L/x windows, the planner reads the study's merge curve (saturation_merge_curve.csv, m = 4, 16, 64, N ≤ 1e7):

• between measured counts, the worse one;
• past 64, the 64 curve;
• with no merge curve for a candidate point, SaturationCurves::load refuses the directory.

Other sketches are unchanged. The main reason is the AutoSketch vs. ASAP evaluation: AutoSketch never merges, so leaving the penalty out favours ASAP.

🤖 Generated with Claude Code

[zzylol]

Milind confirmed on Slack (LGTM). Merging.

🤖 Generated with Claude Code