feat: Implement map-to-string casting

[getChan]

Which issue does this PR close?

Closes #4631.

Rationale for this change

This PR implements native map-to-string casting in Comet to match Spark formatting ({k -> v, ...}).
As part of completing this feature, it also covers the empty-map case (cast(map() as string)) where Catalyst infers Map<NullType, NullType>.

What changes are included in this PR?

• Implement native map-to-string rendering in native/spark-expr.
• Wire Spark-side cast support for MapType -> StringType in CometCast.
• Include empty map support by handling NullType -> StringType in cast support checks.
• Extend codegen NullType handling needed by Map<NullType, NullType>:
  • type admission for NullType
  • output path (NullVector, null-only write/getter handling)
  • input nested read/copy path (NullType getter/copy handling).
• Update SQL test expectation so cast(map() as string) runs natively instead of fallback.

How are these changes tested?

• PROFILES='-Pspark-4.0' make format
• ./mvnw -q test -DwildcardSuites=CometSqlFileTestSuite -Dsuites='org.apache.comet.CometSqlFileTestSuite cast_complex_types_to_string' -Dtest=none

[getChan]

This is intentional. NullType has no typed getter to override (there is no getNull API), so we emit no scalar getter here. Null semantics are handled via isNullAt (always true for NullVector) and generic get(..., NullType) dispatch, which returns null.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

[andygrove]

Note on this review: this was generated by an LLM (Claude Code) at my request while I worked through a review backlog. I have not verified the individual findings myself. Please treat everything below as suggestions to evaluate rather than as authoritative review feedback, and push back on anything that is wrong or already handled.

The rendering matches Spark's {k -> v, ...} and reusing cast_array for the keys and values so nested types compose is the right structure. Flipping a good number of expect_fallback fixtures back to plain query is a satisfying diff to read.

Four things.

Does this honor spark.sql.legacy.castComplexTypesToString.enabled?

cast_map_to_string hardcodes {, }, and " -> ". When that legacy config is on, Spark renders maps with [ and ] and omits NULL elements instead of writing the literal null. CometCast.scala already has a comment about only supporting the non-legacy behavior for structs and arrays, so presumably the same guard needs to cover MapType -> StringType.

Is the map case gated by that check today? If not, a user with the legacy config gets a silently different string. If it is gated, a test with the config on asserting a fallback would pin it.

Hidden entries outside a slice or under a null row

The function casts the entire keys() and values() children up front, then indexes them with the row offsets. For a sliced MapArray, or one with null rows, the children contain entries that the visible rows never reference. Under a throwing eval mode, an ANSI cast failure on one of those hidden values would fail the query even though Spark never evaluates it.

This is the same class of problem #5457 is fixing for nested casts generally. Is it reachable here? If the fix lands in prepare_nested_cast_input at the top of cast_array, it may cover this for free, but it would be worth confirming rather than assuming.

NullType is now admitted to the codegen dispatcher globally

def isSupportedDataType(dt: DataType): Boolean = dt match {
  case NullType => true

That is in CometBatchKernelCodegen, so it admits NullType for every dispatched expression, not just this cast. The supporting changes generate the literal null from elementGetterCall and emitSpecializedGetterExpr, which compiles where an Object is expected and does not where a primitive is.

Is there an expression where a NullType child would land in a primitive context and produce uncompilable Java? A test that dispatches some other expression over a NullType column would be worth having, since the failure mode is a Janino compile error at runtime rather than something CI catches.

The propagation test was inverted

cast MapType propagates Unsupported from nested value cast existed specifically to prove that an unsupported nested cast is not silently swallowed by the outer Map arm. It has been rewritten to assert Compatible for the now-supported case.

That is correct, but it means the original property is no longer covered. Could you add back a case with a genuinely unsupported nested type, for example Map<Int, Binary> to string, asserting that the Unsupported still propagates through the outer arm? Otherwise a future change that swallows the inner result would go unnoticed.

[getChan]

@andygrove thank for review.

Does this honor spark.sql.legacy.castComplexTypesToString.enabled?

Yes. MapType is covered by the existing complex-type legacy guard, so it routes through CodegenDispatchFallback instead of the native map-to-string cast when the config is enabled. I also added an empty-map case to cast_complex_types_to_string_legacy.sql.

[getChan]

Hidden entries outside a slice or under a null row

I added regression tests for both sliced maps and hidden entries under null map rows. As you said, both currently fail. I also verified that both pass when combined with #5457, so this will be resolved once that PR is merged.

[getChan]

NullType is now admitted to the codegen dispatcher globally

I added runtime Janino tests covering a top-level NullType-to-primitive cast, reading an Array element through elementGetterCall in a primitive expression, and writing Array output through emitSpecializedGetterExpr. All three tests compile and pass.

[getChan]

The propagation test was inverted

I restored the Unsupported propagation coverage using Map<Int, CalendarInterval>. The test verifies that the nested CalendarInterval-to-String Unsupported result and reason are propagated through the outer Map arm, and it passes.

The above changes were made with the assistance of an AI agent.