Skip to content

rqe-optimizer: size DDSketch memory from the metric's value range - #152

Merged
zzylol merged 3 commits into
mainfrom
rqe-metric-value-range
Oct 6, 2026
Merged

zzylol merged 3 commits into
mainfrom
rqe-metric-value-range

Conversation

@zzylol

@zzylol zzylol commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Part of #147 (Model side).

  • MetricFacts gets value_range: Option<(f64, f64)>: smallest and largest positive sample value. validate_facts rejects anything but 0 < lo <= hi < inf.
  • For dd rows on a metric with a range, memory per instance is (floor(ln(hi/lo) / ln((1+α)/(1−α))) + 1) × 8 B, with α from sketch_config.params.alpha. This matches dd_footprint in sketch-bench/src/wrappers/dd/mod.rs for positive values (one store, floor + 1 buckets). Without a range, or for other families, the measured mem_bytes_per_instance is kept.
  • One helper, instance_memory_bytes(deployment, facts), replaces the direct reads in ingest, merge and storage memory, and in candidate dominance pruning. Pruning compares KLL and DD candidates for the same quantile RAQE, so it has to use the same size. prune_dominated_candidates now takes facts.
  • small_problem gives the duration metric a 1 ms–60 s range. DD is undeployable by default, so this only matters with --allow-undeployable-families. Docs updated.

Not done: buckets are also capped by values per instance (dd_footprint's min(count)), but facts don't carry that yet. That's the items-per-instance work in #147. The AutoSketch baseline's own resource ranking still uses the table value. Plans it picks are still scored with the range.

Tests: bucket formula, range vs. fallback (no range / non-DD), and validation. cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --check pass.

🤖 Generated with Claude Code

Add optional `MetricFacts::value_range` (lo, hi), validated by
`validate_facts` (0 < lo <= hi < inf). For DDSketch rows with a known
range, memory per instance is (floor(ln(hi/lo) / ln((1+a)/(1-a))) + 1)
buckets x 8 B, matching sketch-bench's `dd_footprint` for positive values
(one store). Without a range, or for other families, the measured
`mem_bytes_per_instance` is kept.

One helper, `instance_memory_bytes`, replaces the direct reads in the
ingest, merge and storage costs and in candidate dominance pruning, which
now takes `facts`.

Part of #147

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er bound

Buckets = min(range buckets, λ·x/card(G)), as dd_footprint caps them by count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol

zzylol commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Grilling outcome (agreed with @zzylol):

  • Q5: the metric-wide range is a conservative upper bound per group; documented.
  • Q6: buckets are now capped by values per instance, λ·x/card(G), as dd_footprint caps by count (new test).
  • Q7: merge CPU stays measured; the size sweep will cover bucket-dependent merge cost.
  • Q8: pruning using the same size, and AutoSketch unchanged, accepted.

🤖 Generated with Claude Code

The accumulator holds L/x windows' values, so its bucket cap is λ·L/card(G).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol merged commit 717f216 into main Oct 6, 2026
2 checks passed
@zzylol

zzylol commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Review fix before merge (e0edca1): the merge accumulator holds the whole lookback, so DDSketch's bucket cap there is λ·L/card(G), not one window's λ·x/card(G). instance_memory_bytes takes the span; merge passes L, ingest/storage/pruning pass x. Test: 5 s windows, 10 s query → 10 values → 10 buckets (was 5).

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant