Skip to content

to_csv never runs inside Comet by default, unlike to_json / from_csv / schema_of_csv #5578

Description

@andygrove

Part of #5572.

to_csv never runs inside Comet by default. Every path through CometStructsToCsv.getSupportLevel returns Unsupported or Incompatible:

if (containsComplexType) {
  return Unsupported(...)
}
if (containsIncompatibleDataTypes) {
  return Incompatible(...)
}
// https://github.com/apache/datafusion-comet/issues/3232
Incompatible()

spark/src/main/scala/org/apache/comet/serde/structs.scala:271-288

The serde has no CodegenDispatchFallback mixin, so with the default configuration the enclosing operator falls back to Spark for every to_csv call. The native path is reachable only with spark.comet.expression.StructsToCsv.allowIncompatible=true.

The rest of the structured-text family does not behave this way. to_json is Hybrid, and from_csv and schema_of_csv are both plain CometCodegenDispatch (serde/csv.scala:24, :26). to_csv is the odd one out.

There is also a docs bug here: docs/source/user-guide/latest/expressions.md lists to_csv as "✅ Native" with no note, which reads as "runs natively by default". It does not. Adding the mixin fixes the docs too, since GenerateDocs derives the Implementation column from the trait mixins and would then report it as Hybrid.

Fix: mix CodegenDispatchFallback into CometStructsToCsv. The complex-type arm dispatches fine — arrays, structs and maps are all in CometBatchKernelCodegen.isSupportedDataType as long as their leaves are.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:expressionsExpression evaluationdocumentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions