Repository navigation
Kafka Connect: Split route field path once instead of per record - #16655
Conversation
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
no stale |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
9eaee5b to
e947c1f
Compare
|
@laskoviymishka you merged #16657 and #16606 in this same |
e947c1f to
b5740d6
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
laskoviymishka
left a comment
There was a problem hiding this comment.
This is a clean win — splitting the route path once in the constructor instead of per record is the right call, and reading tablesRouteField() once is safe since the config is immutable for the writer's lifetime. Behavior looks identical across the null / static / dynamic paths and the existing tests cover it; the one gap is that ROUTE_FIELD is single-segment, so a SinkWriter test with a nested path like a.b would pin the pre-split path end to end — optional.
I'd also reframe the description from a perf fix to a small allocation cleanup: the saving is ~50–130ns against much heavier per-record conversion and write work, and there's no JMH behind the numbers. Nothing here blocks merge.
| private final IcebergWriterFactory writerFactory; | ||
| private final Map<String, RecordWriter> writers; | ||
| private final Map<TopicPartition, Offset> sourceOffsets; | ||
| // the route field is fixed per config, so split its dotted path once instead of per record |
There was a problem hiding this comment.
I'd drop this comment — it describes what the PR changed ("instead of per record"), which stops making sense once the old per-record split is gone, and the field name already says what it holds. If we want a comment here, the one non-obvious bit is the null contract: routeFieldPath == null means route to all tables.
There was a problem hiding this comment.
Dropped in 23e1b00, without a replacement: the null contract is already stated where the field is read, by the // route to all tables branch on the static path and the checkNotNull message on the dynamic one.
| } | ||
|
|
||
| @SuppressWarnings("unchecked") | ||
| static Object extractFromRecordValue(Object recordValue, List<String> fields) { |
There was a problem hiding this comment.
After this change the only production caller goes through the List<String> overload, so the String version is left serving just TestRecordUtils. Either is fine, but I'd pick deliberately: drop the String overload and have the tests split via Splitter.on('.').splitToList(...) so they exercise the production signature, or keep it and treat it as the covered split-then-extract convenience. It's package-private, so there's no API cost either way.
There was a problem hiding this comment.
Dropped the String overload in 23e1b00, and TestRecordUtils now splits with Splitter.on('.').splitToList(...), so it exercises the production signature. Since the split now happens in the SinkWriter constructor, I also added TestSinkWriter.dynamicRouteNestedField, which routes on a nested route.table field and fails if the constructor stops splitting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generated-by: Claude Code (claude-opus-5-5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b5740d6 to
23e1b00
Compare
|
@laskoviymishka both of your comments are addressed in 23e1b00 (replies inline), and CI is green. Could you merge it when you have a moment? Thanks! |
|
Thanks! This is good to merge now |
For static routing with
iceberg.tables.route-fieldand for dynamic routing,SinkWriterre-split the dotted route field path withSplitter.on('.').splitToListon every record, although the route field is fixed for the writer's lifetime. This small allocation cleanup splits the path once in theSinkWriterconstructor and passes the pre-splitList<String>toRecordUtils.extractFromRecordValue. The package-privateStringoverload of that method is removed, soTestRecordUtilsnow exercises the production signature. Behavior is unchanged; a non-JMH microbenchmark put the saving at roughly 50-130 ns per routed record, which is small next to record conversion and writing.Tests: the new
TestSinkWriter.dynamicRouteNestedFieldroutes on a two-segment path (route.table) and fails if the path is not split; the existingTestSinkWriterandTestRecordUtilscases cover both routing modes.AI Disclosure