Skip to content

fix(chart): cap how much memory and how many queries the ClickHouse reader uses at once - #700

Open
reckless-sherixx wants to merge 5 commits into
mainfrom
fix/nb-43217-clickhouse-reader-limits
Open

reckless-sherixx wants to merge 5 commits into
mainfrom
fix/nb-43217-clickhouse-reader-limits

Conversation

@reckless-sherixx

@reckless-sherixx reckless-sherixx commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The read-only ClickHouse user that answers query_data (nudgebee_reader) had a 1.5GB memory limit per query, but no limit on its queries together and none on how many run at once. Two concurrent wide log searches could each take 1.5GB. Together they exceed what the 2000Mi ClickHouse pod has, leaving nothing for the inserts that store logs, traces and metrics.

  • max_memory_usage_for_user (clickhouse.reader.maxMemoryUsageForUser, 1.5GB): all of the reader's running queries together stay inside the room one query used to have.
  • max_memory_usage (clickhouse.reader.maxMemoryUsage) drops from 1.5GB to 1GB, below the per-user cap, so one wide query leaves room for the small trace and picker queries that run next to it.
  • max_concurrent_queries_for_user (clickhouse.reader.maxConcurrentQueries, 8): ClickHouse refuses the next query with TOO_MANY_SIMULTANEOUS_QUERIES instead of running it.
  • Constraints, like the other limits: a query can lower each value but can't raise it or set 0.
  • Safe with --reuse-values: each per-user cap renders only when it's set, so a release upgraded from 0.1.31 runs without them, not with a 0 cap and an unsatisfiable constraint.
  • CI: helm-dev-lint asserts all three caps and their constraints. It also asserts that neither per-user cap renders when unset, while the reader user still does.

Upgrading: helm upgrade doesn't restart ClickHouse, and it reads this users file only at start. Until the pod restarts, an upgraded cluster keeps its old limits. Apply them with kubectl rollout restart statefulset/<release>-clickhouse-shard0. That briefly pauses ingest while the pod restarts. values.yaml says the same.

For reviewers:

  • Cost of the 1GB per-query cap: a single very wide query now fails at 1GB instead of 1.5GB. Sorts and GROUP BYs already spill to disk at 500MB, so this affects mostly wide raw reads.
  • 8 is a starting point, not a measured peak. The runner doesn't limit concurrency itself, so a burst goes straight to ClickHouse and the 9th query is refused at once. The values.yaml comment says to raise it if that error appears under normal use.
  • Merge order: this is now chart 0.1.33, because 0.1.32 is released. feat(chart): make the image registry option move the agent images #697 also bumps the version, so whichever of the two merges second needs the next one.

Type of change

  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change (chart upgrade requires user action)
  • Documentation only
  • CI / tooling

Chart version

  • I bumped version in charts/nudgebee-agent/Chart.yaml (required for user-visible changes): 0.1.33, since 0.1.32 is released.
  • No version bump needed (docs/CI only)

Test plan

  • helm lint / ct lint: run by CI. Helm wasn't available locally.
  • Template rendered with Go text/template, with include stubbed, from the values.yaml defaults. The output parses as XML.
  • helm-dev-lint.yml parses as YAML, the step's script passes bash -n, and its inline Python compiles.
  • ClickHouse 24.12.6.70, with the rendered file in users.d and the container limited to 2000m like the pod:
    • As nudgebee_reader, system.settings shows readonly 2, max_memory_usage 1000000000, max_memory_usage_for_user 1500000000 and max_concurrent_queries_for_user 8.
    • These are each refused with SETTING_CONSTRAINT_VIOLATION:
      • max_concurrent_queries_for_user=100 and =0;
      • max_memory_usage_for_user=3000000000 and =0;
      • max_memory_usage=1500000000 and =0.
    • Lowering both caps is allowed. A query that sets only max_execution_time=30, as the backend does, still runs.
    • Of 9 simultaneous SELECT sleep(3), 8 return and the 9th fails with Code: 202 … Too many simultaneous queries for user nudgebee_reader. Current: 8, maximum: 8.
    • A ~1.3GB range() query fails with Code: 241 … Query memory limit exceeded … maximum: 953.67 MiB. An ~880MB query runs with a ~200MB query alongside it, and both return.
  • Not reproduced: the small-query failure with both caps at 1.5GB. My heavy test query reports only KiB in system.user_processes while it sleeps, so the test couldn't hold memory at the moment the small query ran. The change follows from reading the settings, not from a measured failure.

Related issues

Follows the read-only reader user from #671.

…eader uses at once

The read-only user that answers query_data had a memory limit per query but
none for its queries together, and no limit on how many run at once. Two
concurrent wide log searches could each take their 1.5GB and together exceed
what the 2000Mi ClickHouse pod has, leaving none for the inserts that store
logs, traces and metrics.

- max_memory_usage_for_user (clickhouse.reader.maxMemoryUsageForUser, 1.5GB):
  all of the reader's running queries together get the room one query had.
- max_concurrent_queries_for_user (clickhouse.reader.maxConcurrentQueries, 8):
  ClickHouse refuses the next query with TOO_MANY_SIMULTANEOUS_QUERIES instead
  of running it.
- Both have constraints like the other limits, so a query can lower them but
  neither raise them nor set 0.
- Each is rendered only when set, so a release upgraded with --reuse-values
  from 0.1.31 keeps running without them instead of with a 0 cap.
- helm-dev-lint asserts both caps and their constraints, and that neither
  renders when unset.
- Chart 0.1.32.
@reckless-sherixx
reckless-sherixx requested a review from a team as a code owner November 3, 2026 17:30

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces per-user memory and concurrency limits for the ClickHouse reader user in the Helm chart. Specifically, it adds conditional rendering for max_memory_usage_for_user and max_concurrent_queries_for_user in the user profile and constraints templates, and defines their default values in values.yaml. It also bumps the chart version to 0.1.32. I have no feedback to provide.

@blue4209211
blue4209211 marked this pull request as draft November 5, 2026 03:27
@blue4209211

Copy link
Copy Markdown
Contributor

Revise. The caps are the right way to protect inserts. Two small things before merge: nobody has run them on ClickHouse yet, and the memory cap affects more traffic than the PR says.

Before merge

  1. Run the unchecked ClickHouse 24.12 test. It's the only proof the fix works, and dev has a ClickHouse to run it on. I confirmed both setting names exist on 24.12 (system.settings on dev), but not that the profile and constraints behave as described.
  2. The memory cap can fail small queries too, not just "two heavy reads". nudgebee_reader serves every query_data call: logs, traces, the query service and the SELECT 1 integration test. With the per-user cap equal to the per-query cap (both 1.5GB), one log search near 1.5GB leaves almost nothing, so a small trace or picker query running alongside it can get MEMORY_LIMIT_EXCEEDED (from reading the code, not reproduced). Setting the per-query limit below the per-user one (for example 1GB and 1.5GB) keeps room for those. The cost is that a very wide query fails sooner, though sorts and GROUP BYs already spill at 500MB. If you keep them equal, please state this cost in the "For reviewers" note.

Non-blocking

  1. The caps don't apply on a normal upgrade. Nothing restarts ClickHouse when this ConfigMap changes (no checksum annotation on it), and the file is only read at start. After helm upgrade a cluster stays uncapped until the pod restarts, with no sign of it. This needs a release note at minimum, or a restart trigger if the brief ingest gap is acceptable.
  2. 8 isn't based on measurements. system.query_log is off on dev, and 30 seconds of sampling system.processes caught no reader queries. The runner has no limit of its own, so a burst goes straight to ClickHouse and the 9th query is refused immediately.
  3. Merge order: feat(chart): make the image registry option move the agent images #697 also bumps the chart to 0.1.32, so whichever merges second needs 0.1.33.
Checked and fine
  • --reuse-values: the if guards render nothing when a value is unset, so there's no 0 cap or impossible constraint. CI asserts this.
  • Constraints: same pattern as the existing limits, and min 1 blocks setting 0 (unlimited).
  • Backend queries: they only set max_execution_time, so the new constraints don't refuse them.
  • Scope: inserts run as a different user, so the caps can't block ingestion.
  • Simpler? Limiting concurrency in the runner would miss other clients and wouldn't cap memory. ClickHouse settings are the right layer.
Follow-ups (not for this PR)
  1. Nothing in the backend handles these two errors (Code 202, 241). They'll show as raw failures with no retry, Nubi may simply repeat the query, and the integration test can report ClickHouse as failing during a burst.

… cap

One wide query at the per-query cap left almost nothing of the per-user cap
for the small queries running next to it. The per-query default drops to
1GB under the 1.5GB per-user cap, and CI asserts both. values.yaml says that
helm upgrade doesn't restart ClickHouse, so changed limits need a restart,
and that 8 concurrent queries is a starting point. Chart 0.1.33, since 0.1.32
is released.
@reckless-sherixx

Copy link
Copy Markdown
Contributor Author

Thanks. Changes are in 0645f7f, on top of a merge from main.

Before merge

  1. ClickHouse 24.12 run: I ran it on a local 24.12.6.70 with the rendered users file and a 2000m container limit, rather than on the shared ClickHouse. Results are in the test plan:
    • the settings show the configured values;
    • all six raise-or-zero SETs are refused, and lowering is allowed;
    • the 9th of 9 concurrent queries gets TOO_MANY_SIMULTANEOUS_QUERIES;
    • a query setting only max_execution_time still runs.
  2. Per-query cap: it is now 1GB, under the 1.5GB per-user cap, and CI asserts both. The cost (a very wide read fails sooner) is in "For reviewers". I couldn't reproduce the small-query failure with both caps at 1.5GB, because my test query stops holding memory while it sleeps. So that part rests on reading the settings, and the test plan says so.

Non-blocking

  1. Restart: I went with the note rather than a checksum annotation, so an upgrade doesn't silently pause ingest. There's an "Upgrading" note in the description for the release notes, and the values.yaml comment gives the kubectl rollout restart command.
  2. 8: it's now described as a starting point, not a measured peak, both in values.yaml and in "For reviewers".
  3. Version: the chart is 0.1.33, since 0.1.32 is released. feat(chart): make the image registry option move the agent images #697 needs the next version if it merges after this.

@reckless-sherixx

Copy link
Copy Markdown
Contributor Author

Test results on 0645f7f

The settings, the constraints and the 8-query cap all work, including in the product. But on the default 2000Mi pod, the 1.5GB all-searches cap (maxMemoryUsageForUser) never acts: ClickHouse's own server limit always stops searches first. Everything I changed during testing has been reverted.

Needs a change before merge

  • Lower the defaults. ClickHouse stops itself at 90% of the pod (1.76 GiB), and an empty ClickHouse already holds 430–510 MiB, so searches get about 1.26 GiB, which is less than the 1.40 GiB cap. Of the pairs I tried, 700MB per search / 1.0GB for all searches was the only one where the per-user cap stopped searches. Alternatively, say in values.yaml that both caps must scale with the pod limit, and fix the "leaves room for inserts and merges" comment. Real data raises idle memory: ours idles at about 1.1 GiB.

Non-blocking

  • --reuse-values upgrades get none of the new limits. That's expected, but worth adding to the Upgrading note.
  • The chart accepts a per-search cap above the all-searches cap without warning (9.1).

Not covered: the trace-enricher card (it didn't run during the test window) and the Integrations test-connection button (the agent's built-in ClickHouse doesn't have one).

Setups, every test result, and load numbers

Setups

  • Chart render: helm template and helm lint on the PR branch.
  • Install and upgrade: a local kind cluster. The 0.1.32 baseline is the chart from main (0.1.32 isn't in the helm repo yet). The runner image is a release-like one that includes feat(chart,runner): query_data reads through a read-only ClickHouse user with query limits #671, and the endpoints were set to unreachable addresses.
  • Limits under load: a throwaway ClickHouse on a GKE cluster, using the same image (24.12.4-debian-12-r0-nb-3), a 2000Mi limit and the reader file rendered from this branch.
  • Product: a dev environment's bundled ClickHouse running this branch's reader file for about 30 minutes, then reverted.

Chart (section 1)

Test Result
1.1–1.4 render Pass. Defaults render 1e9 / 1.5e9 / 8 with matching constraints. Setting both per-user values to null drops both lines but keeps readonly 2. 16 / 3e9 render as set. maxConcurrentQueries=0 drops the line, so there's no limit.
1.5 reader disabled Pass. No reader ConfigMap and no CLICKHOUSE_READER_USER. Only the StatefulSet's optional: true volume reference is left, which is harmless.
1.6 CI, version Pass. 0.1.33 / 0.1.33, CI green, helm lint clean.
9.1 per-search 2e9, all-searches 1.5e9 Renders with no warning.

Install and upgrade (section 2, kind)

Test Result
2.1 fresh install Pass. All pods Ready; readonly 2, 1e9 / 1.5e9 / 8.
2.2 plain helm upgrade from 0.1.32 Pass. The ConfigMap has the new values; the ClickHouse pod wasn't restarted (same UID, 0 restarts) and still runs 1.5e9 / 0 / 0.
2.3 rollout restart Pass. 1e9 / 1.5e9 / 8; a raise is refused.
2.4 --reuse-values, then restart Pass. Starts cleanly with 1.5e9 / 0 / 0. The rendered file has no per-user lines, and the per-search cap stays at 1.5e9.
2.5 rollback, then restart Pass. 1.5e9 / 0 / 0.
8.3 runner log Pass. query_data uses the read-only ClickHouse user, and schema setup created 9 tables.

A search can't raise its limits (section 3)

All 9 pass. Each raise or 0 fails with SETTING_CONSTRAINT_VIOLATION; lowering both caps works, and max_execution_time = 30 works.

Concurrency (section 4)

  • 9 simultaneous sleep queries, started inside the pod: exactly one failed with Code: 202 … Current: 8, maximum: 8, after 102 ms. The others took about 5.1 s.
  • 10 admin queries during the burst all succeeded.
  • SELECT 1 as the reader afterwards works.

Memory (section 5, 2000Mi pod)

Queries are uniqExact(number) FROM numbers(N). This query's memory grows in steps (30M → 512 MiB, 40M → 1.00 GiB), so I used those sizes instead of 800MB.

Test Result
5.1 1 GiB search Fails with Query memory limit exceeded … maximum: 953.67 MiB.
5.2 512 MiB search alone Succeeds.
5.3 two 512 MiB searches at once Both succeed.
5.3 three 512 MiB searches at once (3 rounds) Every failure is (total) memory limit exceeded … current RSS 1.76 GiB, maximum: 1.76 GiB. Never User memory limit exceeded.
5.4 GROUP BY over 200M rows Succeeds in 20 s with a 640 MiB peak, so the spill to disk works.

Server ceiling max_server_memory_usage = 1.76 GiB (ratio 0.9). Idle RSS on an empty ClickHouse was 508 MiB.

Writers under load (section 6)

Each run is 5 minutes: an admin insert every ~1 s, while the reader repeatedly runs 4 × 512 MiB searches plus 9 sleep queries.

Per search / all searches Searches stopped by the user cap Stopped by the server limit Inserts failed (of ~275) Peak pod memory
1.5GB / none (0.1.32) – 227 0 1803Mi
1.0GB / 1.5GB (this PR) 0 234 1 1638Mi
700MB / 1.0GB 234 25 1 1786Mi
800MB / 1.2GB ~130 (the other code-241 failures; message text not captured) 120 3 1752Mi
  • There were no ClickHouse restarts or OOMKills in any run.
  • My synthetic writer lost 0–3 inserts in every setting, the old one included, so this test can't show a writer benefit either way. It only shows which limit stops the searches.

Product (section 7–8, dev environment, this branch's file live)

Test Result
Apply and restart Ready in 27 s; ingest per minute didn't dip.
7.1 logs over 1 hour, 24 hours and 7 days Pass: 100 rows each time. A watcher on system.processes showed the searches running as nudgebee_reader; the biggest reached 119 MB.
7.2 trace list and detail during a 7-day log search Pass.
7.3 4 logs tabs at once plus a nubi log question Pass. No refusal, and nubi answered.
7.4 logs screen with all 8 slots held Shows Code: 202 … Too many simultaneous queries for user nudgebee_reader. Current: 8, maximum: 8, not an empty result.
7.5 7 days, all namespaces, |= "error" Shows only "Could not load logs — internal server error". ClickHouse's system.errors shows TIMEOUT_EXCEEDED … maximum: 30 seconds; the search peaked at 419 MB, so the memory caps weren't involved. A clearer timeout message is a backend follow-up, not this PR.
7.6 Integrations test connection N/A. The agent's built-in ClickHouse has no Test Connection button.
8.1 service map Pass. Its queries run as default (admin).
8.2 trace enricher card Not observed: the enricher didn't run during the 30-minute window.
8.4 ingest Pass. 5-minute counts during the test: 365–481k logs and 25–32k traces, against 317–475k and 24–31k in the hour before.

Over the whole window, system.errors showed only the two errors the tests caused on purpose (one TIMEOUT_EXCEEDED, one TOO_MANY_SIMULTANEOUS_QUERIES) and no MEMORY_LIMIT_EXCEEDED.

Operational note for the Upgrading section

Applying the limits needs a ClickHouse restart. On a node where an eBPF uprobe instrumenter attaches to /opt/bitnami/clickhouse/bin/clickhouse (we hit this with SolarWinds OBI), ClickHouse fails its start-up binary checksum check (Code: 246 … CORRUPTED_DATA) and crash-loops. A ClickHouse that is already running is unaffected. Excluding the pod from the instrumenter fixes it, through that tool's exclude annotation or an exe-path exclusion. skip_binary_checksum_checks didn't help on 24.12.4.

@reckless-sherixx
reckless-sherixx marked this pull request as ready for review November 10, 2026 14:55
…se-reader-limits

# Conflicts:
#	.github/workflows/helm-dev-lint.yml
#	charts/nudgebee-agent/templates/clickhouse-reader-users.yaml
#	charts/nudgebee-agent/values.yaml
… a 2000Mi pod

On the default 2000Mi ClickHouse, the server's own limit (90% of the pod)
minus ClickHouse's idle memory left less than the 1.5GB per-user cap, so
the server limit always failed searches first and the per-user cap never
acted. Lower the defaults to 700MB per query / 1.0GB per user, the pair
that stopped searches at the per-user cap under load, and say in
values.yaml that both caps must scale with the pod limit.

The chart now refuses a per-query cap above the per-user cap, and CI
asserts both the new defaults and the refusal. Chart 0.1.34.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants