Skip to content

Avoid CometColumnarExchange when next query stage requires CometColumnarToRow #2880

Closed

Description

@andygrove

What is the problem the feature request solves?

The golden file at spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q1.native_datafusion/extended.txt has this section:

      :     :     :  +- HashAggregate
      :     :     :     +- CometColumnarToRow
      :     :     :        +- CometColumnarExchange
      :     :     :           +- HashAggregate

Presumably the first HashAggregate shown here is not supported by Comet (fallback reasons are missing for some reason) so this query stage needs to call CometColumnarToRow.

There is probably no benefit in using columnar shuffle in the child stage if the parent stage immediately converts back to row-based, although we should confirm that assumption.

Describe the potential solution

Additional context

No response

Activity

  1. added this to the 0.13.0 milestone on Dec 11, 2025
  2. modified the milestones: 0.13.0, 0.14.0 on Dec 12, 2025
  3. removed this from the 0.14.0 milestone on Mar 12, 2026
  4. andygrove commented on Sep 28, 2026

    @andygrove
    MemberAuthor

    This is the same shape as #4004, which #4010 fixed by reverting a Comet columnar shuffle to a Spark shuffle when both its parent and child are Spark hash aggregates. The pattern no longer appears in any of the TPC-DS golden plans, and the broader question of when Comet should decline a stage is tracked in #4005.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions