Skip to content

KafkaInputReader: header/key values win over payload on name collision - #505

Open
ravisharm wants to merge 1 commit into
34.0.0-confluentfrom
kafka-input-header-wins
Open

ravisharm wants to merge 1 commit into
34.0.0-confluentfrom
kafka-input-header-wins

Conversation

@ravisharm

Copy link
Copy Markdown
Member

Summary

KafkaInputReader: Kafka header/key/topic values now win over payload columns of the same name.

 buildBlendedEventMap.get(key)
-  val = payloadRow.get(key)
-  if val == null -> headerMap.get(key)
+  val = headerMap.get(key)
+  if val == null -> payloadRow.get(key)

Applies to read() and sample(). Needed so enrichment headers (T0, Metrics Monetization) can't be shadowed by a payload column.

Evidence

  • Before: header foo=fromHeader + payload foo=x, empty header prefix
    testHeaderWinsOnNameCollision -> ComparisonFailure: expected:<fromHeader> but was:<x>
    After: KafkaInputFormatTest Tests run: 12, Failures: 0, Errors: 0

Note: the Kafka module has no junit-vintage-engine, so its JUnit 4 tests run 0 tests under Surefire by default. I added the engine temporarily to run them and did not commit that.

Merge Danger

Door: two-way

Revert the 4-line change.

Blast Radius: Kafka ingestion

Behavior changes only for specs using KafkaInputFormat with a header, key or topic column name that equals a payload column. With the default kafka.header. prefix, collisions are rare. Anyone relying on payload-wins would now see header values.

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@ravisharm
ravisharm requested a review from a team as a code owner October 19, 2026 07:21
Copilot AI balanced review requested due to automatic review settings October 19, 2026 07:21

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused change matches the intended precedence, with no blocking issues identified in static review; tests were not executed.

No open issues 🎉

What changed in this PR

Updates Kafka ingestion so non-null header, key, and topic values take precedence over same-named payload columns.

Changes:

  • Reverses shared lookup precedence for read() and sample().
  • Adds a header-collision regression test that also checks unaffected payload fields and timestamps.
  • Updates the lookup documentation.
File Description
extensions-core/​kafka-indexing-service/​src/​test/​java/​org/​apache/​druid/​data/​input/​kafkainput/​KafkaInputFormatTest.java Tests header precedence on name collisions.
extensions-core/​kafka-indexing-service/​src/​main/​java/​org/​apache/​druid/​data/​input/​kafkainput/​KafkaInputReader.java Prioritizes Kafka metadata over payload values and documents the behavior.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants