Repository navigation
fix(chart): cap how much memory and how many queries the ClickHouse reader uses at once - #700
reckless-sherixx wants to merge 5 commits into
Conversation
…eader uses at once The read-only user that answers query_data had a memory limit per query but none for its queries together, and no limit on how many run at once. Two concurrent wide log searches could each take their 1.5GB and together exceed what the 2000Mi ClickHouse pod has, leaving none for the inserts that store logs, traces and metrics. - max_memory_usage_for_user (clickhouse.reader.maxMemoryUsageForUser, 1.5GB): all of the reader's running queries together get the room one query had. - max_concurrent_queries_for_user (clickhouse.reader.maxConcurrentQueries, 8): ClickHouse refuses the next query with TOO_MANY_SIMULTANEOUS_QUERIES instead of running it. - Both have constraints like the other limits, so a query can lower them but neither raise them nor set 0. - Each is rendered only when set, so a release upgraded with --reuse-values from 0.1.31 keeps running without them instead of with a 0 cap. - helm-dev-lint asserts both caps and their constraints, and that neither renders when unset. - Chart 0.1.32.
There was a problem hiding this comment.
Code Review
This pull request introduces per-user memory and concurrency limits for the ClickHouse reader user in the Helm chart. Specifically, it adds conditional rendering for max_memory_usage_for_user and max_concurrent_queries_for_user in the user profile and constraints templates, and defines their default values in values.yaml. It also bumps the chart version to 0.1.32. I have no feedback to provide.
|
Revise. The caps are the right way to protect inserts. Two small things before merge: nobody has run them on ClickHouse yet, and the memory cap affects more traffic than the PR says. Before merge
Non-blocking
Checked and fine
Follow-ups (not for this PR)
|
… cap One wide query at the per-query cap left almost nothing of the per-user cap for the small queries running next to it. The per-query default drops to 1GB under the 1.5GB per-user cap, and CI asserts both. values.yaml says that helm upgrade doesn't restart ClickHouse, so changed limits need a restart, and that 8 concurrent queries is a starting point. Chart 0.1.33, since 0.1.32 is released.
|
Thanks. Changes are in 0645f7f, on top of a merge from Before merge
Non-blocking
|
Test results on 0645f7fThe settings, the constraints and the 8-query cap all work, including in the product. But on the default 2000Mi pod, the 1.5GB all-searches cap ( Needs a change before merge
Non-blocking
Not covered: the trace-enricher card (it didn't run during the test window) and the Integrations test-connection button (the agent's built-in ClickHouse doesn't have one). Setups, every test result, and load numbersSetups
Chart (section 1)
Install and upgrade (section 2, kind)
A search can't raise its limits (section 3)All 9 pass. Each raise or 0 fails with Concurrency (section 4)
Memory (section 5, 2000Mi pod)Queries are
Server ceiling Writers under load (section 6)Each run is 5 minutes: an admin insert every ~1 s, while the reader repeatedly runs 4 × 512 MiB searches plus 9
Product (section 7–8, dev environment, this branch's file live)
Over the whole window, Operational note for the Upgrading sectionApplying the limits needs a ClickHouse restart. On a node where an eBPF uprobe instrumenter attaches to |
…se-reader-limits # Conflicts: # .github/workflows/helm-dev-lint.yml # charts/nudgebee-agent/templates/clickhouse-reader-users.yaml # charts/nudgebee-agent/values.yaml
… a 2000Mi pod On the default 2000Mi ClickHouse, the server's own limit (90% of the pod) minus ClickHouse's idle memory left less than the 1.5GB per-user cap, so the server limit always failed searches first and the per-user cap never acted. Lower the defaults to 700MB per query / 1.0GB per user, the pair that stopped searches at the per-user cap under load, and say in values.yaml that both caps must scale with the pod limit. The chart now refuses a per-query cap above the per-user cap, and CI asserts both the new defaults and the refusal. Chart 0.1.34.
Summary
The read-only ClickHouse user that answers
query_data(nudgebee_reader) had a 1.5GB memory limit per query, but no limit on its queries together and none on how many run at once. Two concurrent wide log searches could each take 1.5GB. Together they exceed what the 2000Mi ClickHouse pod has, leaving nothing for the inserts that store logs, traces and metrics.max_memory_usage_for_user(clickhouse.reader.maxMemoryUsageForUser, 1.5GB): all of the reader's running queries together stay inside the room one query used to have.max_memory_usage(clickhouse.reader.maxMemoryUsage) drops from 1.5GB to 1GB, below the per-user cap, so one wide query leaves room for the small trace and picker queries that run next to it.max_concurrent_queries_for_user(clickhouse.reader.maxConcurrentQueries, 8): ClickHouse refuses the next query withTOO_MANY_SIMULTANEOUS_QUERIESinstead of running it.--reuse-values: each per-user cap renders only when it's set, so a release upgraded from 0.1.31 runs without them, not with a 0 cap and an unsatisfiable constraint.helm-dev-lintasserts all three caps and their constraints. It also asserts that neither per-user cap renders when unset, while the reader user still does.Upgrading:
helm upgradedoesn't restart ClickHouse, and it reads this users file only at start. Until the pod restarts, an upgraded cluster keeps its old limits. Apply them withkubectl rollout restart statefulset/<release>-clickhouse-shard0. That briefly pauses ingest while the pod restarts.values.yamlsays the same.For reviewers:
8is a starting point, not a measured peak. The runner doesn't limit concurrency itself, so a burst goes straight to ClickHouse and the 9th query is refused at once. Thevalues.yamlcomment says to raise it if that error appears under normal use.Type of change
Chart version
versionincharts/nudgebee-agent/Chart.yaml(required for user-visible changes): 0.1.33, since 0.1.32 is released.Test plan
helm lint/ct lint: run by CI. Helm wasn't available locally.text/template, withincludestubbed, from thevalues.yamldefaults. The output parses as XML.helm-dev-lint.ymlparses as YAML, the step's script passesbash -n, and its inline Python compiles.users.dand the container limited to 2000m like the pod:nudgebee_reader,system.settingsshowsreadonly2,max_memory_usage1000000000,max_memory_usage_for_user1500000000 andmax_concurrent_queries_for_user8.SETTING_CONSTRAINT_VIOLATION:max_concurrent_queries_for_user=100and=0;max_memory_usage_for_user=3000000000and=0;max_memory_usage=1500000000and=0.max_execution_time=30, as the backend does, still runs.SELECT sleep(3), 8 return and the 9th fails withCode: 202 … Too many simultaneous queries for user nudgebee_reader. Current: 8, maximum: 8.range()query fails withCode: 241 … Query memory limit exceeded … maximum: 953.67 MiB. An ~880MB query runs with a ~200MB query alongside it, and both return.system.user_processeswhile it sleeps, so the test couldn't hold memory at the moment the small query ran. The change follows from reading the settings, not from a measured failure.Related issues
Follows the read-only reader user from #671.