Support DELTA_BINARY_PACKED and DELTA_BYTE_ARRAY

[kazuyukitanimura]

What is the problem the feature request solves?

There are some tests in Spark 4.0 that uses parquet.writer.version=v2 (ParquetTypeWideningSuite).

The V2 write writes with delta encoding. Comet currently cannot read such files

Describe the potential solution

No response

Additional context

No response

[parthchandra]

IIRC, the vectorized versions of these encodings in Spark did not improve performance much over the row based implementation in the parquet library

[andygrove]

Closing as superseded: the new native_datafusion and native_iceberg_compat scans (default spark.comet.scan.impl=auto) read DELTA_BINARY_PACKED / DELTA_BYTE_ARRAY / DELTA_LENGTH_BYTE_ARRAY via arrow-rs. Please reopen if there is still a need to support these encodings in the legacy native_comet scan.