Skip to content

feat(chart): upgrade the bundled ClickHouse to 26.10 - #730

Open
TechNinjaAyush wants to merge 2 commits into
mainfrom
feat/clickhouse-26.10
Open

TechNinjaAyush wants to merge 2 commits into
mainfrom
feat/clickhouse-26.10

Conversation

@TechNinjaAyush

Copy link
Copy Markdown
Contributor

Summary

Upgrades the bundled ClickHouse from 24.12 to the official clickhouse/clickhouse-server:26.10.4.207 image.

Upgrading is one-way. 26.10 reads a volume that 24.12 wrote. Once 26.10 has written to it, 24.12 can no longer start on it (Code: 79. Unknown mark file extension). Snapshot the ClickHouse volume before upgrading. To roll back, restore the snapshot; setting the old tag again won't work.

Two config additions:

  • skip_binary_checksum_checks=1: on nodes where eBPF agents attach uprobes, ClickHouse's start-up checksum check of its own binary fails (Code: 246, CORRUPTED_DATA), and the server crash-loops on restart. We reproduced this on a test cluster. With the setting, 9/9 restarts succeeded.
  • 24.12 defaults kept for two settings 26 changed: async_insert=0, so collector inserts are confirmed only once written, and output_format_json_quote_64bit_integers=1 for both the default and nudgebee_reader profiles, so the JSON returned to readers is unchanged.

Type of change

  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change (chart upgrade requires user action): snapshot the ClickHouse volume first; no downgrade without restoring it
  • Documentation only
  • CI / tooling

Chart version

  • I bumped version in charts/nudgebee-agent/Chart.yaml (required for user-visible changes)
  • No version bump needed: the release commit bumps it

Test plan

  • helm lint charts/nudgebee-agent passes
  • ct lint --config .github/configs/ct.yaml passes (left to CI)
  • helm template output reviewed: compared with main, only the ClickHouse image (pod and schema job), the two config entries and the config checksum change
  • Installed on a real/kind cluster and verified the change:
    • Ran ClickHouse 26.10.4.207 in Docker with the rendered config. system.server_settings shows skip_binary_checksum_checks=1. system.settings_profile_elements shows async_insert=0 and output_format_json_quote_64bit_integers=1 on default, and readonly=2 plus quoted integers on nudgebee_reader. The reader user logs in.
    • Upgraded a dev cluster from 24.12 to the same image digest (sha256:4f491abc…597035). All 17 table definitions are identical. Row counts and checksums over a fixed 2-hour window match before and after. All tables and materialized views keep ingesting, and the collector resumed writing within seconds of the restart.

To verify as a reviewer:

helm template t charts/nudgebee-agent --show-only templates/clickhouse-configmap.yaml | grep -E "skip_binary|async_insert|quote_64bit"
helm template t charts/nudgebee-agent --show-only templates/clickhouse-statefulset.yaml | grep image:

Related issues

Part of nudgebee/nudgebee-enterprise#40165

Run the official clickhouse/clickhouse-server:26.10.4.207 image. 26.10
reads volumes 24.12 wrote; the reverse does not hold, so roll back by
restoring a volume snapshot, not by setting the older tag.

- skip_binary_checksum_checks=1: on nodes where eBPF agents attach
  uprobes, the start-up binary checksum check fails (code 246) and the
  server crash-loops on restart.
- Keep the 24.12 behaviour for two defaults 26 changed: async_insert=0
  and output_format_json_quote_64bit_integers=1 (default and
  nudgebee_reader profiles).
@TechNinjaAyush
TechNinjaAyush requested a review from a team as a code owner November 16, 2026 10:36

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request upgrades ClickHouse to version 26.10.4.207, disables binary checksum checks to prevent crash-loops with eBPF agents, and introduces compatibility settings to preserve ClickHouse 24.12 behaviors. Feedback suggests moving the nudgebee_reader profile configuration to clickhouse-reader-users.yaml to avoid splitting configuration files. Additionally, the reviewer warns that switching the registry from ghcr.io to docker.io may lead to Docker Hub rate limiting and the loss of custom security patches, suggesting instead to mirror the new image on ghcr.io.

Comment on lines +133 to +135
<nudgebee_reader>
<output_format_json_quote_64bit_integers>1</output_format_json_quote_64bit_integers>
</nudgebee_reader>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The nudgebee_reader profile is only used when the read-only reader user is enabled (defined in clickhouse-reader-users.yaml). Splitting the configuration of the nudgebee_reader profile across two different files (10_compat_settings.xml and nudgebee-reader.xml) makes the configuration harder to maintain and debug.

Instead of defining the nudgebee_reader profile here, you can remove it from 10_compat_settings.xml and add the <output_format_json_quote_64bit_integers>1</output_format_json_quote_64bit_integers> setting directly to the nudgebee_reader profile definition in charts/nudgebee-agent/templates/clickhouse-reader-users.yaml.

Comment on lines +684 to +686
registry: docker.io
repository: clickhouse/clickhouse-server
tag: 26.10.4.207

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Switching the registry from ghcr.io to docker.io and using the official unpatched image can introduce two main issues:

  1. Docker Hub Rate Limiting: Anonymous image pulls from Docker Hub are subject to strict rate limits (100 pulls per 6 hours per IP address). In Kubernetes environments where multiple nodes or clusters share a NAT gateway IP, this can frequently lead to ImagePullBackOff errors during deployments or scaling events.
  2. Security Patches: The previous image (ghcr.io/nudgebee/clickhouse) was mirrored with OS patches. Using the official image directly means losing those custom OS-level vulnerability patches.

Consider maintaining a mirrored version of the 26.10.4.207 image on ghcr.io (e.g., ghcr.io/nudgebee/clickhouse:26.10.4.207-nb-1) to ensure production reliability and security.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant