Skip to content

Benchmark nested types as native shuffle hash partitioning keys #5787

Description

@viirya

What is the problem the feature request solves?

spark.comet.shuffle.native.partitioning.hash.nested.enabled (#5567) lets a struct, array or
map be a native shuffle hash partitioning key. Nothing in CometShuffleBenchmark measures those
shapes, so there is no baseline for whether native hashing of a nested key beats letting Spark do
the shuffle, and no way to see a regression in that path.

The existing cases do not cover it:

  • shuffleArrayBenchmark and shuffleStructBenchmark repartition on a nested column but have no
    native-shuffle case, because the gate rejected nested keys when they were written.
  • shuffleDeeplyNestedBenchmark calls repartition(n) with no key, which is round robin rather
    than hash partitioning, so it never exercises the hash kernel.

This is separate from #2904, which was about columnar/JVM shuffle throughput on nested schemas
with a keyless repartition(n).

Describe the potential solution

Add a shuffleNestedHashKeyBenchmark with Spark, Comet (Spark Shuffle), Comet (JVM Shuffle) and
Comet (Native Shuffle) cases, where the native case enables the nested config so it measures the
native hashing path rather than a silent fallback to Spark's shuffle.

Cover shapes that separate the two code paths in the native Murmur3 kernel: a list whose elements
are primitives is vectorized, while a list whose elements are themselves nested falls through to
hash_list_array!. So struct<int, string>, array<int>, struct<array<int>, string>,
array<struct<int, string>> and struct<map<string, int>, int>, at a small and a large partition
count.

Additional context

Measuring this is what surfaced #5777: array<struct<..>> as a native hash key runs at 0.3-0.4x
of leaving the shuffle to Spark, while every other shape measured is faster natively. The
benchmark should land independently of any kernel change so the before/after comparison has a
reference point that is not moving at the same time.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions