Skip to content

Map lookups with float, collated or complex keys fall back to Spark (map_col[key], element_at) #5580

Description

@andygrove

Part of #5572.

MapKeySupport.keySupport (spark/src/main/scala/org/apache/comet/serde/maps.scala:68) returns Unsupported for three classes of map key:

  • floating point anywhere in the key type, because Spark normalizes -0.0 to +0.0 and canonicalizes NaN in ArrayBasedMapBuilder while native map_extract compares raw Arrow values
  • a non-default collation, because native compares as UTF8_BINARY
  • any complex key type, because map_extract's coercion casts the lookup key to the map's exact Arrow key type

Two serdes consume it, and neither mixes in CodegenDispatchFallback:

  • CometMapExtract (GetMapValue, i.e. map_col[key]) — serde/maps.scala:117, gate at :120
  • CometElementAt on map input — serde/arrays.scala:614

So map_col[key] and element_at(map_col, key) fail the whole projection back to Spark whenever the map has double, collated-string or struct keys.

The analysis behind each decline is right — these really are cases the native lookup gets wrong — which is exactly why the dispatcher is the correct answer rather than a native fix. Spark's own GetMapValue.doGenCode and ElementAt.doGenCode give the normalization and interpreted-ordering equality for free, and MapType is fully supported by CometBatchKernelCodegen.isSupportedDataType (recursively, as long as the leaves are).

Fix: mix CodegenDispatchFallback into both serdes.

Note that CometElementAt's other Unsupported arm, "Input must be an array or map", is unreachable — Spark's own type checking rejects that before Comet sees it — so it needs no handling.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions