Skip to content

lpad / rpad with a non-literal pad argument falls back to Spark #5579

Description

@andygrove

Part of #5572.

lpad and rpad decline two shapes, and both fall the whole projection back to Spark:

if (expr.str.isInstanceOf[Literal]) {
  return Unsupported(Some(PadReasons.literalStrReason))
}
if (!expr.pad.isInstanceOf[Literal]) {
  return Unsupported(Some(PadReasons.nonLiteralPadReason))
}

spark/src/main/scala/org/apache/comet/serde/strings.scala:413-419 (CometStringRPad) and :441-447 (CometStringLPad)

The non-literal pad restriction is the one that matters. lpad(name, 10, pad_col) — padding with a value that varies per row — is a perfectly ordinary query shape, and there is nothing about it that Spark's generated code cannot do. StringRPad and StringLPad are plain codegen expressions over StringType and IntegerType, all of which CometBatchKernelCodegen.isSupportedDataType admits.

The literal-str restriction is lower value (ConstantFolding handles most of it) but costs nothing extra to cover.

Fix: mix CodegenDispatchFallback into both serdes.

Worth checking while in here: docs/source/user-guide/latest/expressions.md renders lpad and rpad with an em-dash in the Implementation column, meaning GenerateDocs found no serde registered under those function names. Either the registry name differs from the function name or the rows need fixing — either way the doc currently says nothing useful about how these execute.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions