[EPIC] Faster Spark-to-Arrow conversion in CometSparkToColumnarExec

[andygrove]

What / Why

CometSparkToColumnarExec converts Spark rows and Spark column vectors to Arrow so that native operators can consume them. The conversion goes through ArrowWriter, which CometLocalTableScanExec and the Comet cache serializer also use. I measured it one column type at a time on main at 93d1189be, with 8192-row batches. Several common cases are far slower than they need to be:

• Decimals cost 30-160 ns per row, 30 to 100 times the cost of an int. Arrow's BigDecimal setter allocates a BigInteger and two byte arrays per value, on top of the Decimal and BigDecimal that Spark's getters build.
• The bulk copy for fixed-width columns (perf: bulk copy fixed-width columns in ArrowWriter #5442) is skipped when a batch holds a single null, and for every dictionary-encoded column. Spark's Parquet reader leaves low-cardinality columns dictionary-encoded.
• Strings cost 13-16 ns per row, written one value at a time through Arrow's setSafe. A dictionary-encoded string also allocates a copy of its bytes on every row, because Parquet's dictionary returns a fresh array for each decode.
• Arrays, maps and structs are written one element at a time: about 90 ns per row for array<string> and 200 ns for map<string,string>, with five entries on average.
• Spark's vectorized Parquet reader produces 4096-row batches, and each becomes one Arrow batch, so native operators get half of Comet's batch size. The cache produces 10000-row batches, which become alternating Arrow batches of 8192 and 1808 rows.

End to end, conversion took 549 ms of a 1088 ms TPC-H Q1-style query over Spark's Parquet reader.

Work

• Write Spark columns to Arrow in bulk for decimals, nullable and dictionary-encoded fixed-width columns, booleans, strings, arrays, maps and structs, and fill each Arrow batch across Spark batches: perf: write Spark columns to Arrow in bulk in CometSparkToColumnarExec #6566 (merged).
• Reuse Arrow buffers across batches in Spark-to-Arrow conversion instead of reallocating and zeroing them #6720 Reallocating and zeroing every buffer for each batch (allocateNew) is about half the remaining cost of the fixed-width columnar path once the above lands.
• Write strings and nested values in bulk when converting Spark rows to Arrow #6721 Row input still writes strings and nested values one at a time. The failure semantics of the row-path string writer are pinned by the tests from perf: copy off-heap strings directly into Arrow #6281.
• Admit all array and map types in CometSparkToColumnarExec #6722 CometSparkToColumnarExec.isTypeSupported still declines arrays other than ARRAY<STRING> and maps other than MAP<STRING,STRING>, so those columns never reach the conversion and the operators above them stay on Spark. In the feat: admit string maps in Spark-to-Comet conversion #6036 review, dropping that gate passed every probe query.
• Row-input conversionTime metric includes the time the child spends producing rows #6723 For row input, the conversionTime metric includes the time the child spends producing rows, so it overstates the conversion.
• Stop ending Spark-to-Arrow batches at input file boundaries #6752 SparkColumnarArrowReader ends each Arrow batch at an input file boundary. That protects nothing once fix: keep a Spark scan unconverted when the plan reads input_file_name #6703 makes the conversion fall back for input_file_name, and it leaves small batches when a task reads many small files.