Skip to content

Driver warns that spark.executor.memoryOverhead is unset when spark.executor.memoryOverheadFactor is set, and in local mode #6189

Description

@andygrove

Describe the bug

CometDriverPlugin.warnIfExecutorMemoryOverheadUnset warns whenever spark.executor.memoryOverhead is not set (Plugins.scala#L165-L180). For large executors, the tuning guide recommends raising spark.executor.memoryOverheadFactor instead (tuning.md#L177-L184). Users who follow that advice are still told to set spark.executor.memoryOverhead.

The warning also fires in local mode, where there is no container to size. The native memory usage log already skips local mode.

Steps to reproduce

Start an application with the Comet plugin, off-heap memory enabled, spark.executor.memoryOverheadFactor=0.2 and no spark.executor.memoryOverhead. The driver logs "spark.executor.memoryOverhead is not set".

Expected behavior

No warning when the overhead is configured through spark.executor.memoryOverheadFactor or spark.kubernetes.memoryOverheadFactor, or when the master is local.

Activity

  1. added this to the 1.1.0 milestone on Sep 24, 2026
  2. self-assigned this
    on Sep 24, 2026
  3. added a commit that references this issue on Sep 25, 2026
    2c41b95
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:memoryMemory pools, reservations, OOM handlingbugSomething isn't workingrequires-triage

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions