Skip to content

Report native operator spill metrics in Spark task metrics for non-shuffle stages #5447

Description

@peterxcli

What / Why

#5370 and #5445 bridge native spill metrics into Spark task metrics (diskBytesSpilled / memoryBytesSpilled), but only for native shuffle write tasks: CometMetricNode.reportSpillMetrics is registered solely on the CometNativeShuffleInputRDD path.

A spill-capable native operator running in a non-shuffle stage (e.g. a sort in a result stage, executed via CometExecRDD in operators.scala) still reports spilled_bytes only as a SQL operator metric. The Spark Stages/task view shows zero spill for the task, the same SQL-view vs task-view discrepancy that #5382 fixed for unified shuffle plans.

Proposed scope

  • Register a task-level spill reporting callback for native stages executed via CometExecRDD, aggregating over the stage's CometMetricNode tree as sumMetricValues does for shuffle tasks (shared accumulators counted once, disk and memory kept separate, no memory value inferred from disk bytes).
  • Register it before the native iterator so final metrics are published first, including for failed attempts, matching the ordering established in fix: report native child spill metrics in shuffle tasks #5445.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions