Skip to content

Explain ObjectHashAggregate fallback when Comet shuffle is disabled #5500

Description

@sunchao

Describe the bug

CometObjectHashAggregate returns None when Comet shuffle is disabled without attaching a fallback reason. This is a pre-existing diagnostics gap identified while reviewing #5421, not a regression from that PR's aggregate-buffer repair.

The shuffle guard in the reviewed source bypasses doConvert and does not call withFallbackReason. The strict unexplained-fallback check can therefore report an unannotated decline, or a reason collected from another node can obscure this particular cause.

Steps to reproduce

Add a focused serde/planner regression for ObjectHashAggregateExec, such as grouped collect_list over an eligible Parquet input with spark.sql.execution.useObjectHashAggregateExec=true and spark.comet.shuffle.enabled=false. Check the shuffle-disabled guard directly and the resulting extended-explain reason; run the regression with spark.comet.explain.fallback.strict.enabled both enabled and disabled.

The missing annotation is verified from the guard's source. A separate end-to-end strict-mode reproduction has not been run for this issue.

Expected behavior

The deliberate decline should carry an accurate shuffle-disabled reason, without changing aggregate execution eligibility or masking an unrelated missing explanation.

Additional context

The review discussion explicitly separates this issue from the buffer-compatibility fix. #5499 tracks stale fallback-reason provenance; this issue covers a specific missing reason at its source.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:aggregationHash aggregates, aggregate expressionsbugSomething isn't workingpriority:lowMinor issues, test failures, tooling, cosmetic

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions