Repository navigation
[VL] Spark 4.1: Support Parquet struct field compatibility improvements #11914
Description
Backend
VL (Velox)
Bug description
Two Parquet struct field handling issues in GlutenParquetIOSuite on Spark 4.1:
Issue 1: SPARK-53535 — Spark 4.1 added returnNullStructIfAllFieldsMissing parameter for missing struct field handling. Gluten has not adapted to this change.
Issue 2: Vectorized reader missing struct fields — Pre-existing test (SPARK-34863) for vectorized reader complex type handling. SPARK-53535 modified this test, causing it to fail on Spark 4.1 with Gluten.
These are two independent root causes but both only affect Spark 4.1.
Parent issue: #11550
Impact
| Suite | Exclude | spark40 | spark41 |
|---|---|---|---|
| GlutenParquetIOSuite | SPARK-53535 | 🟢 | 🔴 |
| GlutenParquetIOSuite | vectorized reader: missing all struct fields | 🟢 | 🔴 |
Note: GlutenParquetIOSuite also has a SPARK-54220 (NullType) exclude tracked separately under the Tracking issue #11910.
References
- Apache Spark JIRA: SPARK-53535
- Apache Spark JIRA: SPARK-34863
Activity
rui-mo commentedon Jun 9, 2026
The behaviors mentioned in this patch are supposed to be enabled through the changes in Velox PR #17555, which Gluten will soon depend on. Thanks.