Repository navigation
rqe-optimizer: size DDSketch memory from the metric's value range - #152
Merged
Merged
Conversation
Add optional `MetricFacts::value_range` (lo, hi), validated by `validate_facts` (0 < lo <= hi < inf). For DDSketch rows with a known range, memory per instance is (floor(ln(hi/lo) / ln((1+a)/(1-a))) + 1) buckets x 8 B, matching sketch-bench's `dd_footprint` for positive values (one store). Without a range, or for other families, the measured `mem_bytes_per_instance` is kept. One helper, `instance_memory_bytes`, replaces the direct reads in the ingest, merge and storage costs and in candidate dominance pruning, which now takes `facts`. Part of #147 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 tasks
…er bound Buckets = min(range buckets, λ·x/card(G)), as dd_footprint caps them by count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Grilling outcome (agreed with @zzylol):
🤖 Generated with Claude Code |
The accumulator holds L/x windows' values, so its bucket cap is λ·L/card(G). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Review fix before merge (e0edca1): the merge accumulator holds the whole lookback, so DDSketch's bucket cap there is 🤖 Generated with Claude Code |
This was referenced Oct 6, 2026
Merged
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #147 (Model side).
MetricFactsgetsvalue_range: Option<(f64, f64)>: smallest and largest positive sample value.validate_factsrejects anything but0 < lo <= hi < inf.ddrows on a metric with a range, memory per instance is(floor(ln(hi/lo) / ln((1+α)/(1−α))) + 1) × 8 B, with α fromsketch_config.params.alpha. This matchesdd_footprintinsketch-bench/src/wrappers/dd/mod.rsfor positive values (one store,floor + 1buckets). Without a range, or for other families, the measuredmem_bytes_per_instanceis kept.instance_memory_bytes(deployment, facts), replaces the direct reads in ingest, merge and storage memory, and in candidate dominance pruning. Pruning compares KLL and DD candidates for the same quantile RAQE, so it has to use the same size.prune_dominated_candidatesnow takesfacts.small_problemgives the duration metric a 1 ms–60 s range. DD is undeployable by default, so this only matters with--allow-undeployable-families. Docs updated.Not done: buckets are also capped by values per instance (
dd_footprint'smin(count)), but facts don't carry that yet. That's the items-per-instance work in #147. The AutoSketch baseline's own resource ranking still uses the table value. Plans it picks are still scored with the range.Tests: bucket formula, range vs. fallback (no range / non-DD), and validation.
cargo test --workspace,cargo clippy --workspace --all-targets -- -D warningsandcargo fmt --checkpass.🤖 Generated with Claude Code