test: move the Spark 4 collation suites to Comet SQL tests #6633

Part of #6615.

What / Why

This issue moves the Spark 4 collation suites to Comet SQL tests:

• spark-4.0/CometCollationSuite
• spark-4.1/CometCollationSuite
• spark-4.x/CometSortCollationSuite
• spark-4.x/CometCastCollatedStringSuite

Static analysis on main at 3bc2faa found the following:

The 4.1 collation suite is the 4.0 suite minus its five join-converter tests, so the other 20 tests are identical in both. Converted into fixtures gated -- MinSparkVersion: 4.0, they run on 4.0, 4.1 and 4.2, which takes most of the duplication that #4770 is about off the table.

It also adds coverage. The shuffle and sort fallback reasons have never been checked on Spark 4.2, because the 4.0 and 4.1 suites don't run there. If 4.2 reports different reasons, split the fixture into a MaxSparkVersion: 4.1 / MinSparkVersion: 4.2 pair.

Collated tests in other suites (rlike, unix_timestamp, string concat/reverse) are in #6627, #6625 and #6626.

Facts that affect the conversion

1. Collated literals. 'x' COLLATE c parses to Collate(Literal, ...). With ConstantFolding excluded it stays a Collate node, which has no Comet serde. CAST('MON' AS STRING COLLATE UTF8_LCASE) works instead, because CometCast folds a cast of a literal into a collated Literal.
   • That is enough for next_day's native path (4.0 L386 / 4.1 L207).
   • It is not enough for date_format, because CometDateFormat.nativeApplicable needs a Spark-level Literal format. So 4.0 L403 / 4.1 L224 wait on test: let Comet SQL tests keep ConstantFolding, exclude optimizer rules and change configs mid-file #6618.
2. Collated datetime fallbacks record two reasons. When a CodegenDispatchFallback serde reports Incompatible for a collation and the dispatcher is disabled, Comet records both <fn>: spark.comet.exec.scalaUDF.codegen.enabled=false and ... does not support non-UTF8_BINARY collations. routing_datetime_collation_disabled.sql asserts the first and the Scala tests assert the second, so each expression needs a second expect_fallback query.
3. Cast eval modes. For CometCastCollatedStringSuite, the cast values never raise. So -- ConfigMatrix: spark.sql.ansi.enabled=false,true plus try_cast covers all three eval modes that isSupported checks.

Work

Collation-suite line numbers are given as "4.0 Lx / 4.1 Ly".

Datetime collation routing (existing fixtures)

• datetime/routing_datetime_collation_disabled.sql
  • Add a second query per expression asserting the collation-specific reason: next_day (L292 / L113), convert_timezone (L359 / L180), trunc (L366 / L187), date_trunc (L372 / L193), and date_format with a collated column format (L379 / L200).
  • Add from_unixtime with expect_fallback(Only the default datetime format pattern) (L315 / L136).
  • Add expect_native(unix_timestamp) for unix_timestamp(ts, fmt_ts COLLATE UTF8_LCASE) (L299 / L120), using a new fmt_ts column.
• datetime/routing_datetime_collation_enabled.sql
  • The same unix_timestamp query with codegen on.
  • expect_dispatch(date_format) with a collated column format (L426 / L247). The existing query uses a foldable Collate(Literal) format, which takes DateFormatClass's precomputed-formatter branch instead of the per-row one.
• datetime/unix_timestamp.sql (L440 / L261): add the two-argument unix_timestamp(ts, fmt_column) with expect_native.

New fixtures (all MinSparkVersion: 4.0)

• datetime/next_day_collation_allow_incompatible.sql (L386 / L207)
  • Codegen off and NextDay.allowIncompatible=true.
  • expect_native(next_day) over next_day(d, CAST('MON' AS STRING COLLATE UTF8_LCASE)).
• string/collation_shuffle_fallback.sql (L47/L35, L55/L43, L62/L50, L70/L58, L77/L65)
  • listagg DISTINCT, DISTINCT and GROUP BY on a collated string, each with expect_fallback(unsupported hash partitioning data type for columnar shuffle).
  • ORDER BY on a collated string, with expect_fallback(unsupported range partitioning data type for columnar shuffle).
  • A native control over Parquet.
• windows/window_sort_collation.sql (sort L51, L118, and the window half of L182), with AQE off
  • expect_fallback(Sort does not support non-default string collation) for keys c, _1, _1, c, _2 and struct(array(c)), _2.
  • Native window controls.
• string/collation_sort.sql (sort L160 and the TopK half of L182): TopK ORDER BY c, _1 LIMIT 2 over a REPARTITION(1) subquery.
• string/collation_sort_round_robin.sql (sort L135): jvm shuffle, with SELECT /*+ REPARTITION(2) */ ... SORT BY c, _1.
• cast/routing_cast_collation_disabled.sql and cast/routing_cast_collation_enabled.sql (CometCastCollatedStringSuite)
  • Table (_1 string, _2 int) with ('123', 1) and ('456', 2), plus the ANSI ConfigMatrix.
  • Disabled file: codegen off, and expect_fallback(Cast involving a non-default string collation is not supported) for L98, L102, L109, L113, L117, L131, L140, L161, L170, L174, the convertible halves of L146, L151 and L156, and the end-to-end tests L224 and L243.
  • Enabled file: expect_dispatch(cast) for the same expressions, the end-to-end tests L234 and L261, and expect_native(cast) for the uncollated half of L183.
  • Things to check:
    • L109: Spark must type AS STRING as UTF8_BINARY.
    • L151 and L170: Spark 4 must accept a collated map key in CreateMap.
    • L174: the suite says no SQL builds ArrayType(NullType) -> ARRAY<STRING COLLATE UTF8_LCASE>, but with ConstantFolding excluded CAST(array(NULL) AS ...) should reach isSupported.

Blocked

• 4.0 L403 / 4.1 L224 (date_format with a collated literal format): needs ConstantFolding, test: let Comet SQL tests keep ConstantFolding, exclude optimizer rules and change configs mid-file #6618.

Stays in Scala

• 4.0 L119, L149, L175, L203, L228 call the join operator serde on hand-built plans; the test comment says end-to-end SQL cannot reach the join converter. The 4.1 copy dropped these, although they build plans directly and could run on 4.1+. Consider moving them to a 4.x shim while deleting the rest of these suites.
• Sort L65, L93 call QueryPlanSerde.supportedSortType on synthetic SortExec plans: map keys and values, which SQL cannot sort by, and the struct single-key versus multi-key rule.
• 4.0 L328 / 4.1 L149 pin explain text (fix: remove misleading native opt-in for dispatch-only datetime expressions #6182). Their dispatch/fallback assertions duplicate routing_datetime_collation_{enabled,disabled}.sql L57-61.
• The identity casts in CometCastCollatedStringSuite (L121, L135, L189, and the identity halves of L146, L151, L156 and L183). SimplifyCasts drops them before Comet sees them.

Done when

• Every test in the table below is accounted for. Either a fixture covers it (the same inputs, configs and assertion strength, or stronger), the fixture that already covers it is cited, or it stays in Scala for the reason given above.
• Each converted Scala test is deleted in the PR that adds its coverage. A suite left empty is deleted and removed from the suite lists in .github/workflows/pr_build_linux.yml and pr_build_macos.yml.
• The new and changed fixtures pass on every Spark profile. Apply run-all-spark-profiles, because pull requests run only the default profile.

What stays in Scala afterwards (the explain-text tests and the join-converter serde tests) is small enough to move into one spark-4.x suite. That would retire both version-specific CometCollationSuite files and resolve #4770.