What is the problem the feature request solves?
spark.comet.shuffle.native.partitioning.hash.nested.enabled (#5567) lets a struct, array or
map be a native shuffle hash partitioning key. Nothing in CometShuffleBenchmark measures those
shapes, so there is no baseline for whether native hashing of a nested key beats letting Spark do
the shuffle, and no way to see a regression in that path.
The existing cases do not cover it:
shuffleArrayBenchmark and shuffleStructBenchmark repartition on a nested column but have no
native-shuffle case, because the gate rejected nested keys when they were written.
shuffleDeeplyNestedBenchmark calls repartition(n) with no key, which is round robin rather
than hash partitioning, so it never exercises the hash kernel.
This is separate from #2904, which was about columnar/JVM shuffle throughput on nested schemas
with a keyless repartition(n).
Describe the potential solution
Add a shuffleNestedHashKeyBenchmark with Spark, Comet (Spark Shuffle), Comet (JVM Shuffle) and
Comet (Native Shuffle) cases, where the native case enables the nested config so it measures the
native hashing path rather than a silent fallback to Spark's shuffle.
Cover shapes that separate the two code paths in the native Murmur3 kernel: a list whose elements
are primitives is vectorized, while a list whose elements are themselves nested falls through to
hash_list_array!. So struct<int, string>, array<int>, struct<array<int>, string>,
array<struct<int, string>> and struct<map<string, int>, int>, at a small and a large partition
count.
Additional context
Measuring this is what surfaced #5777: array<struct<..>> as a native hash key runs at 0.3-0.4x
of leaving the shuffle to Spark, while every other shape measured is faster natively. The
benchmark should land independently of any kernel change so the before/after comparison has a
reference point that is not moving at the same time.
What is the problem the feature request solves?
spark.comet.shuffle.native.partitioning.hash.nested.enabled(#5567) lets a struct, array ormap be a native shuffle hash partitioning key. Nothing in
CometShuffleBenchmarkmeasures thoseshapes, so there is no baseline for whether native hashing of a nested key beats letting Spark do
the shuffle, and no way to see a regression in that path.
The existing cases do not cover it:
shuffleArrayBenchmarkandshuffleStructBenchmarkrepartition on a nested column but have nonative-shuffle case, because the gate rejected nested keys when they were written.
shuffleDeeplyNestedBenchmarkcallsrepartition(n)with no key, which is round robin ratherthan hash partitioning, so it never exercises the hash kernel.
This is separate from #2904, which was about columnar/JVM shuffle throughput on nested schemas
with a keyless
repartition(n).Describe the potential solution
Add a
shuffleNestedHashKeyBenchmarkwith Spark, Comet (Spark Shuffle), Comet (JVM Shuffle) andComet (Native Shuffle) cases, where the native case enables the nested config so it measures the
native hashing path rather than a silent fallback to Spark's shuffle.
Cover shapes that separate the two code paths in the native Murmur3 kernel: a list whose elements
are primitives is vectorized, while a list whose elements are themselves nested falls through to
hash_list_array!. Sostruct<int, string>,array<int>,struct<array<int>, string>,array<struct<int, string>>andstruct<map<string, int>, int>, at a small and a large partitioncount.
Additional context
Measuring this is what surfaced #5777:
array<struct<..>>as a native hash key runs at 0.3-0.4xof leaving the shuffle to Spark, while every other shape measured is faster natively. The
benchmark should land independently of any kernel change so the before/after comparison has a
reference point that is not moving at the same time.