Repository navigation
[CORE][SPARK-4.0] Only require one config spark.plugins to enable gluten #5915
Description
Description
For now, there are at least three required configs to enable gluten:
- spark.plugins
- spark.memory.offHeap.enabled
- spark.shuffle.manager
Less required configs is always better. Fortunately, We can set spark.memory.offHeap.enabled and spark.shuffle.manager inside GlutenDriverPlugin since Spark 4.0.0, so only spark.plugins is required.
- spark.memory.offHeap.enabled: [SPARK-46947][CORE] Delay memory manager initialization until Driver plugin is loaded spark#45052
- spark.shuffle.manager: [SPARK-45762][CORE] Support shuffle managers defined in user jars by changing startup order spark#43627
Activity
[-]Only require one config spark.plugins to enable gluten[/-][+][CORE][SPARK-4.0] Only require one config spark.plugins to enable gluten[/+]FelixYBW commentedon May 29, 2024
We may set shuffle manager default as local shuffle manager. We may overwrite offheap.enabled=true once offheap.size is set.
acvictor commentedon Jun 3, 2024
@FelixYBW as long as ColumnarShuffleManager is in the classpath, do you see any issues with setting default shuffle manager to ColumnarShuffleManager?
FelixYBW commentedon Jun 3, 2024
No. go ahead to submit a PR.
acvictor commentedon Jun 4, 2024
Where would this be set? I think in Spark 3.x it must be set as part of session builder?