Skip to content

Default Kafka ClickPipes to no auth when no auth flags are given - #629

Open
sdairs wants to merge 1 commit into
fix/605-clickpipe-scale-usage-errorfrom
fix/606-kafka-no-auth
Open

Default Kafka ClickPipes to no auth when no auth flags are given#629
sdairs wants to merge 1 commit into
fix/605-clickpipe-scale-usage-errorfrom
fix/606-kafka-no-auth

Conversation

@sdairs

@sdairs sdairs commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

What

clickpipe create kafka and clickpipe schema-discover kafka implicitly defaulted the authentication mechanism to PLAIN whenever --auth was omitted. An invocation with no auth flags at all therefore failed client-side with PLAIN requires --username and --password, making a broker that requires no authentication unreachable.

Auth resolution is now:

  1. an explicit --auth wins;
  2. otherwise the mechanism is inferred from the credential flags that were passed — --username/--password → PLAIN, --access-key-id/--secret-key → IAM_USER, --iam-role → IAM_ROLE, --client-certificate/--client-key → MUTUAL_TLS;
  3. otherwise no authentication is sent.

An explicitly selected mechanism still fails fast when its credentials are missing, and the error now names the mechanism the user actually asked for rather than always saying PLAIN.

Why this request shape

An unauthenticated Kafka source has no value in the spec enum (PLAIN..MUTUAL_TLS), and ClickPipePostKafkaSource carries no required[], so absence of the key is the wire representation of "no auth":

  • the object-storage source already works this way — its authentication is nullable and omitted for public buckets ("PUBLIC uses no authentication" in the spec description), and the CLI already omits it;
  • the equivalent PATCH property (ClickPipePatchKafkaSource.authentication) is nullable upstream;
  • the control plane's own field is omitempty (visible in the official Terraform provider's internal/api/clickpipe_models.go).

So ClickPipePostKafkaSource.authentication becomes Option<T> with skip_serializing_if, paired with a documented optionality_exemptions entry in the analyzer config. Presence and enum-value checking are unchanged; only the requiredness demand is exempted. credentials stays as-is and carries null for a no-auth source, exactly as it already does for IAM_ROLE.

Tests

  • Builder unit tests: no-auth default, each inference path (PLAIN / IAM_USER / IAM_ROLE / MUTUAL_TLS), explicit-mechanism error messages for PLAIN/SCRAM/IAM_USER/MUTUAL_TLS, build_kafka_credentials(None, ..) → null.
  • Inference tests for half-specified credential pairs (no mechanism inferred).
  • Clap coverage: schema-discover kafka parses with no auth flags and builds a source with no authentication.
  • Wiremock subprocess tests: create kafka and schema-discover kafka with no auth flags omit authentication and send null credentials; --username/--password without --auth still sends PLAIN with the right credential body.
  • Library test pinning that an absent mechanism is omitted rather than serialized as null.
  • Verified locally: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test -p clickhousectl, cargo test -p clickhouse-cloud-api --test spec_coverage_test --test models_test --lib, cargo test -p clickhouse-openapi-analyzer, cargo check --workspace --all-features. Cloud integration suites were not run (they need live credentials); the Kafka stages/smoke constructions were updated to the new Option field and compile under --all-targets.

README documents the optional --auth behaviour with a no-auth example for both commands.

Part of a stacked PR chain: this PR is based on fix/605-clickpipe-scale-usage-error, not main.

Fixes #606

🤖 Generated with Claude Code

@sdairs
sdairs force-pushed the fix/606-kafka-no-auth branch from 9013172 to e7a45b4 Compare August 28, 2026 14:46

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e7a45b4. Configure here.

Comment thread crates/clickhousectl/src/cloud/clickpipes.rs
…given

`clickpipe create kafka` and `clickpipe schema-discover kafka` defaulted the
authentication mechanism to PLAIN whenever `--auth` was omitted, so an
invocation with no auth flags at all failed client-side with
"PLAIN requires --username and --password" and a broker that requires no
authentication was unreachable.

An unauthenticated Kafka source has no value in the spec enum
(PLAIN..MUTUAL_TLS), and the schema carries no `required[]`; absence of the
key is the wire representation, matching the object-storage source (whose
`authentication` is already omitted for public buckets) and the control
plane's own `omitempty` field. `ClickPipePostKafkaSource.authentication` is
therefore now `Option<T>` with `skip_serializing_if`, plus a documented
`optionality_exemptions` entry so the drift analyzer keeps checking presence
and enum values without demanding a mechanism per request.

The CLI now resolves the mechanism as: explicit `--auth` wins; otherwise it is
inferred from the credential flags that were passed
(`--username`/`--password` → PLAIN, `--access-key-id`/`--secret-key` →
IAM_USER, `--iam-role` → IAM_ROLE, `--client-certificate`/`--client-key` →
MUTUAL_TLS); otherwise no authentication is sent. An explicitly selected
mechanism still fails fast when its credentials are missing, and the message
now names the mechanism the user actually asked for instead of always PLAIN.

Tests: builder unit tests for the no-auth default, each inference path, and
the explicit-mechanism error messages; inference tests for half-specified
credential pairs; clap coverage that schema-discover parses with no auth flags
and builds a no-auth source; wiremock subprocess tests asserting `create` and
`schema-discover` omit `authentication` and send null `credentials` with no
auth flags, and still send PLAIN when only `--username`/`--password` are
given; a library test pinning that an absent mechanism is omitted rather than
serialized as `null`.

Fixes #606

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sdairs
sdairs force-pushed the fix/606-kafka-no-auth branch from e7a45b4 to fc98375 Compare August 28, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

clickpipe create kafka / schema-discover kafka reject no-auth brokers with implicit PLAIN error

1 participant