Skip to content

Batch ingestion: preserve Arrow columnar batches beyond the Iceberg reader #20401

Open

Description

@Shekharrajak

Description

PR #19510 added an opt-in Iceberg Arrow reader, but the native batch-ingestion boundary is still row-oriented. IcebergArrowInputSourceReader currently converts every Arrow ColumnarBatch row into a Map<String, Object> and MapBasedInputRow. InputSourceReader then exposes CloseableIterator<InputRow>, InputSourceProcessor loops over individual rows, and IncrementalIndex.add accepts one InputRow at a time.

This issue proposes carrying the columnar representation deeper into native batch ingestion so Arrow decoding gains are not lost to per-row materialization before indexing.

Motivation

Local JMH results after #19510 show the bottleneck moving downstream:

  • Reader-only workloads are 1.23–1.58x faster, with a 1.36x geomean.
  • End-to-end read -> incremental index -> segment persist is 1.02x faster .

Acceptance criteria

  • Existing row-based input sources and ingestion specifications remain compatible.
  • The batch path demonstrates a statistically significant end-to-end improvement on at least one representative wide-table workload without regressing the TPC-H lineitem case.
  • Unsupported operations fall back clearly to the row path rather than changing results.

Activity

  1. Shekharrajak commented on Sep 24, 2026

    @Shekharrajak
    ContributorAuthor

    I am looking into this. The expensive part is ColumnarBatch -> HashMap -> MapBasedInputRow .
    For every row, the current Iceberg reader - create hashamap, project values and dimensions and create MapBasedInputRow for druid segment.

    We must have something like ColumnarBatch -> RowsAndColumns -> Batch-backed InputRow -> Persist Druid segment

  2. FrankChen021 commented on Sep 26, 2026

    @FrankChen021
    Member

    We must have something like ColumnarBatch -> RowsAndColumns -> Batch-backed InputRow -> Persist Druid segment

    I think the direction is correct, we need to process input rows in batched columns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions