Skip to content

Kafka Connect: Split route field path once instead of per record - #16655

Merged
laskoviymishka merged 2 commits into
apache:mainfrom
wombatu-kun:kafka-connect-presplit-route-field
Oct 12, 2026
Merged

laskoviymishka merged 2 commits into
apache:mainfrom
wombatu-kun:kafka-connect-presplit-route-field

Conversation

@wombatu-kun

@wombatu-kun wombatu-kun commented Jun 2, 2026 •

Copy link
Copy Markdown
Contributor

For static routing with iceberg.tables.route-field and for dynamic routing, SinkWriter re-split the dotted route field path with Splitter.on('.').splitToList on every record, although the route field is fixed for the writer's lifetime. This small allocation cleanup splits the path once in the SinkWriter constructor and passes the pre-split List<String> to RecordUtils.extractFromRecordValue. The package-private String overload of that method is removed, so TestRecordUtils now exercises the production signature. Behavior is unchanged; a non-JMH microbenchmark put the saving at roughly 50-130 ns per routed record, which is small next to record conversion and writing.

Tests: the new TestSinkWriter.dynamicRouteNestedField routes on a two-segment path (route.table) and fails if the path is not split; the existing TestSinkWriter and TestRecordUtils cases cover both routing modes.


AI Disclosure

  • Model: Claude Opus 4.8, Claude Opus 5.5
  • Platform/Tool: Claude Code
  • Human Oversight: fully reviewed
  • Prompt Summary: Split the Kafka Connect route field path once at construction instead of per record, then address review feedback.

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Jul 3, 2026
@wombatu-kun

Copy link
Copy Markdown
Contributor Author

no stale

@github-actions github-actions Bot removed the stale label Jul 4, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Aug 10, 2026
@wombatu-kun
wombatu-kun force-pushed the kafka-connect-presplit-route-field branch from 9eaee5b to e947c1f Compare August 10, 2026 09:58
@github-actions github-actions Bot removed the stale label Aug 11, 2026
@wombatu-kun

Copy link
Copy Markdown
Contributor Author

@laskoviymishka you merged #16657 and #16606 in this same connect/data package - would you have a moment for this one? Same shape as #16657: per-record work hoisted out of the hot path, 2 files, CI green.

@wombatu-kun
wombatu-kun force-pushed the kafka-connect-presplit-route-field branch from e947c1f to b5740d6 Compare September 10, 2026 04:01
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Oct 11, 2026

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a clean win — splitting the route path once in the constructor instead of per record is the right call, and reading tablesRouteField() once is safe since the config is immutable for the writer's lifetime. Behavior looks identical across the null / static / dynamic paths and the existing tests cover it; the one gap is that ROUTE_FIELD is single-segment, so a SinkWriter test with a nested path like a.b would pin the pre-split path end to end — optional.

I'd also reframe the description from a perf fix to a small allocation cleanup: the saving is ~50–130ns against much heavier per-record conversion and write work, and there's no JMH behind the numbers. Nothing here blocks merge.

private final IcebergWriterFactory writerFactory;
private final Map<String, RecordWriter> writers;
private final Map<TopicPartition, Offset> sourceOffsets;
// the route field is fixed per config, so split its dotted path once instead of per record

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd drop this comment — it describes what the PR changed ("instead of per record"), which stops making sense once the old per-record split is gone, and the field name already says what it holds. If we want a comment here, the one non-obvious bit is the null contract: routeFieldPath == null means route to all tables.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 23e1b00, without a replacement: the null contract is already stated where the field is read, by the // route to all tables branch on the static path and the checkNotNull message on the dynamic one.

}

@SuppressWarnings("unchecked")
static Object extractFromRecordValue(Object recordValue, List<String> fields) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After this change the only production caller goes through the List<String> overload, so the String version is left serving just TestRecordUtils. Either is fine, but I'd pick deliberately: drop the String overload and have the tests split via Splitter.on('.').splitToList(...) so they exercise the production signature, or keep it and treat it as the covered split-then-extract convenience. It's package-private, so there's no API cost either way.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped the String overload in 23e1b00, and TestRecordUtils now splits with Splitter.on('.').splitToList(...), so it exercises the production signature. Since the split now happens in the SinkWriter constructor, I also added TestSinkWriter.dynamicRouteNestedField, which routes on a nested route.table field and fails if the constructor stops splitting.

Vova Kolmakov and others added 2 commits October 12, 2026 09:08
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generated-by: Claude Code (claude-opus-5-5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wombatu-kun
wombatu-kun force-pushed the kafka-connect-presplit-route-field branch from b5740d6 to 23e1b00 Compare October 12, 2026 02:57
@wombatu-kun

Copy link
Copy Markdown
Contributor Author

@laskoviymishka both of your comments are addressed in 23e1b00 (replies inline), and CI is green. Could you merge it when you have a moment? Thanks!

@laskoviymishka
laskoviymishka merged commit 561a4e3 into apache:main Oct 12, 2026
24 of 25 checks passed
@laskoviymishka

Copy link
Copy Markdown
Contributor

Thanks! This is good to merge now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants