Skip to content

Native sequence raises "Unreachable code reached." where Spark 4.0.5, 4.1.4 and 4.2.1 return the sequence (SPARK-58821) #6769

Open

Description

@andygrove

Describe the bug

Spark 4.0.5, 4.1.4 and 4.2.1 include SPARK-58821. When stop - start overflows Long but the
sequence is short enough to allocate, Sequence now returns the recomputed length instead of
raising internalError("Unreachable code reached."). Comet's native sequence kernel
(array_funcs/sequence.rs) reproduces the old error for parity, so on those releases Comet fails a
query that Spark answers.

Steps to reproduce

SELECT sequence(-9223372036854775808L, 9223372036854775807L, 9223372036854775807L)

Against the staged 4.1.4-rc2, 4.0.5-rc1 and 4.2.1-rc1 artifacts, expressions/array/sequence.sql
fails because Spark no longer raises for its expect_error(Unreachable code reached) case. main
at f80042f fails the same way.

Expected behavior

Return the sequence on releases with SPARK-58821, and keep the error on earlier ones. The version
can be resolved in Scala and passed to the kernel.

Additional context

Found by running the array suites against the Spark release candidates for #6563.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:expressionsExpression evaluationbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions