[VL] Spark 4.1: Support memory shuffle spill by size threshold (SPARK-49386) #11922

Backend

VL (Velox)

Gluten version: main branch

Description

Spark 4.1 introduced memory-based shuffle spill thresholds (SPARK-49386, JIRA type: Improvement). The new spillSizeThreshold parameter enables spilling by data size rather than only by row count. Gluten's shuffle implementation does not support this threshold.

Spark 4.1 only.

Parent issue: #11910 ([VL] Spark 4.x: Tracking new feature support)

Impact

Note: GlutenSQLWindowFunctionSuite has a pre-existing spill issue ("low buffer spill threshold") unrelated to SPARK-49386 — out of scope for this issue.

References

• Apache Spark JIRA: SPARK-49386