Describe the bug
CometDriverPlugin.warnIfExecutorMemoryOverheadUnset warns whenever spark.executor.memoryOverhead is not set (Plugins.scala#L165-L180). For large executors, the tuning guide recommends raising spark.executor.memoryOverheadFactor instead (tuning.md#L177-L184). Users who follow that advice are still told to set spark.executor.memoryOverhead.
The warning also fires in local mode, where there is no container to size. The native memory usage log already skips local mode.
Steps to reproduce
Start an application with the Comet plugin, off-heap memory enabled, spark.executor.memoryOverheadFactor=0.2 and no spark.executor.memoryOverhead. The driver logs "spark.executor.memoryOverhead is not set".
Expected behavior
No warning when the overhead is configured through spark.executor.memoryOverheadFactor or spark.kubernetes.memoryOverheadFactor, or when the master is local.
Describe the bug
CometDriverPlugin.warnIfExecutorMemoryOverheadUnsetwarns wheneverspark.executor.memoryOverheadis not set (Plugins.scala#L165-L180). For large executors, the tuning guide recommends raisingspark.executor.memoryOverheadFactorinstead (tuning.md#L177-L184). Users who follow that advice are still told to setspark.executor.memoryOverhead.The warning also fires in local mode, where there is no container to size. The native memory usage log already skips local mode.
Steps to reproduce
Start an application with the Comet plugin, off-heap memory enabled,
spark.executor.memoryOverheadFactor=0.2and nospark.executor.memoryOverhead. The driver logs "spark.executor.memoryOverhead is not set".Expected behavior
No warning when the overhead is configured through
spark.executor.memoryOverheadFactororspark.kubernetes.memoryOverheadFactor, or when the master is local.