You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #19510 added an opt-in Iceberg Arrow reader, but the native batch-ingestion boundary is still row-oriented. IcebergArrowInputSourceReader currently converts every Arrow ColumnarBatch row into a Map<String, Object> and MapBasedInputRow. InputSourceReader then exposes CloseableIterator<InputRow>, InputSourceProcessor loops over individual rows, and IncrementalIndex.add accepts one InputRow at a time.
This issue proposes carrying the columnar representation deeper into native batch ingestion so Arrow decoding gains are not lost to per-row materialization before indexing.
Motivation
Local JMH results after #19510 show the bottleneck moving downstream:
Reader-only workloads are 1.23–1.58x faster, with a 1.36x geomean.
End-to-end read -> incremental index -> segment persist is 1.02x faster .
Acceptance criteria
Existing row-based input sources and ingestion specifications remain compatible.
The batch path demonstrates a statistically significant end-to-end improvement on at least one representative wide-table workload without regressing the TPC-H lineitem case.
Unsupported operations fall back clearly to the row path rather than changing results.
I am looking into this. The expensive part is ColumnarBatch -> HashMap -> MapBasedInputRow .
For every row, the current Iceberg reader - create hashamap, project values and dimensions and create MapBasedInputRow for druid segment.
We must have something like ColumnarBatch -> RowsAndColumns -> Batch-backed InputRow -> Persist Druid segment
Activity
Shekharrajak commentedon Sep 24, 2026
I am looking into this. The expensive part is
ColumnarBatch -> HashMap -> MapBasedInputRow.For every row, the current Iceberg reader - create hashamap, project values and dimensions and create MapBasedInputRow for druid segment.
We must have something like
ColumnarBatch -> RowsAndColumns -> Batch-backed InputRow -> Persist Druid segmentFrankChen021 commentedon Sep 26, 2026
I think the direction is correct, we need to process input rows in batched columns.