Repository navigation
feat(chart): upgrade the bundled ClickHouse to 26.10 - #730
TechNinjaAyush wants to merge 2 commits into
Conversation
Run the official clickhouse/clickhouse-server:26.10.4.207 image. 26.10 reads volumes 24.12 wrote; the reverse does not hold, so roll back by restoring a volume snapshot, not by setting the older tag. - skip_binary_checksum_checks=1: on nodes where eBPF agents attach uprobes, the start-up binary checksum check fails (code 246) and the server crash-loops on restart. - Keep the 24.12 behaviour for two defaults 26 changed: async_insert=0 and output_format_json_quote_64bit_integers=1 (default and nudgebee_reader profiles).
There was a problem hiding this comment.
Code Review
This pull request upgrades ClickHouse to version 26.10.4.207, disables binary checksum checks to prevent crash-loops with eBPF agents, and introduces compatibility settings to preserve ClickHouse 24.12 behaviors. Feedback suggests moving the nudgebee_reader profile configuration to clickhouse-reader-users.yaml to avoid splitting configuration files. Additionally, the reviewer warns that switching the registry from ghcr.io to docker.io may lead to Docker Hub rate limiting and the loss of custom security patches, suggesting instead to mirror the new image on ghcr.io.
| <nudgebee_reader> | ||
| <output_format_json_quote_64bit_integers>1</output_format_json_quote_64bit_integers> | ||
| </nudgebee_reader> |
There was a problem hiding this comment.
The nudgebee_reader profile is only used when the read-only reader user is enabled (defined in clickhouse-reader-users.yaml). Splitting the configuration of the nudgebee_reader profile across two different files (10_compat_settings.xml and nudgebee-reader.xml) makes the configuration harder to maintain and debug.
Instead of defining the nudgebee_reader profile here, you can remove it from 10_compat_settings.xml and add the <output_format_json_quote_64bit_integers>1</output_format_json_quote_64bit_integers> setting directly to the nudgebee_reader profile definition in charts/nudgebee-agent/templates/clickhouse-reader-users.yaml.
| registry: docker.io | ||
| repository: clickhouse/clickhouse-server | ||
| tag: 26.10.4.207 |
There was a problem hiding this comment.
Switching the registry from ghcr.io to docker.io and using the official unpatched image can introduce two main issues:
- Docker Hub Rate Limiting: Anonymous image pulls from Docker Hub are subject to strict rate limits (100 pulls per 6 hours per IP address). In Kubernetes environments where multiple nodes or clusters share a NAT gateway IP, this can frequently lead to
ImagePullBackOfferrors during deployments or scaling events. - Security Patches: The previous image (
ghcr.io/nudgebee/clickhouse) was mirrored with OS patches. Using the official image directly means losing those custom OS-level vulnerability patches.
Consider maintaining a mirrored version of the 26.10.4.207 image on ghcr.io (e.g., ghcr.io/nudgebee/clickhouse:26.10.4.207-nb-1) to ensure production reliability and security.
Summary
Upgrades the bundled ClickHouse from 24.12 to the official
clickhouse/clickhouse-server:26.10.4.207image.Upgrading is one-way. 26.10 reads a volume that 24.12 wrote. Once 26.10 has written to it, 24.12 can no longer start on it (
Code: 79. Unknown mark file extension). Snapshot the ClickHouse volume before upgrading. To roll back, restore the snapshot; setting the old tag again won't work.Two config additions:
skip_binary_checksum_checks=1: on nodes where eBPF agents attach uprobes, ClickHouse's start-up checksum check of its own binary fails (Code: 246, CORRUPTED_DATA), and the server crash-loops on restart. We reproduced this on a test cluster. With the setting, 9/9 restarts succeeded.async_insert=0, so collector inserts are confirmed only once written, andoutput_format_json_quote_64bit_integers=1for both thedefaultandnudgebee_readerprofiles, so the JSON returned to readers is unchanged.Type of change
Chart version
versionincharts/nudgebee-agent/Chart.yaml(required for user-visible changes)Test plan
helm lint charts/nudgebee-agentpassesct lint --config .github/configs/ct.yamlpasses (left to CI)helm templateoutput reviewed: compared withmain, only the ClickHouse image (pod and schema job), the two config entries and the config checksum changesystem.server_settingsshowsskip_binary_checksum_checks=1.system.settings_profile_elementsshowsasync_insert=0andoutput_format_json_quote_64bit_integers=1ondefault, andreadonly=2plus quoted integers onnudgebee_reader. The reader user logs in.sha256:4f491abc…597035). All 17 table definitions are identical. Row counts and checksums over a fixed 2-hour window match before and after. All tables and materialized views keep ingesting, and the collector resumed writing within seconds of the restart.To verify as a reviewer:
Related issues
Part of nudgebee/nudgebee-enterprise#40165