test: move regex coverage to Comet SQL tests and fix fixtures whose patterns lose their backslashes #6627

Part of #6615.

What / Why

This issue moves the regex coverage in CometRegExpJvmSuite, CometRegexParitySuite and the rlike tests in CometExpressionSuite to Comet SQL tests under sql-tests/expressions/string/. Static analysis on main at 3bc2faa found the following:

Only 14 of the 95 are covered by a fixture today, and none of those depends on \d, \w or a backreference. The reason is the first item below.

Fix first: seven regex fixtures don't run the patterns they name

Spark's SQL string-literal unescaping (unescapeSQLString, with spark.sql.parser.escapedStringLiterals=false) drops the backslash before any character that is not a recognized escape. So '\d' becomes d, '\w' becomes w, '\1' becomes 1 and '\+' becomes +.

These fixtures write single backslashes:

Examples:

• rlike_java.sql:27 runs ^d+$.
• The "backreference (Java-only)" queries run ^(w)1$, which contains no backreference.
• regexp_replace_java.sql:34 ('(\d+)-(\d+)') matches no row, so its $2-$1 replacement is never applied.
• rlike_auto_native.sql:38 ('a\+b') is a+b, i.e. one or more a followed by b, not the literal string.

The two *_enabled.sql regex fixtures already write '\\d' correctly. Use '\\d' or a raw literal r'\d'; raw literals work on every supported Spark version.

The Scala tests have the same bug. In CometExpressionSuite's rlike test (L1507), none of the classes and anchors (\d \D \w \W \b \B \h \H \s \S \v \V \A \Z \z) ever reached a regex engine. In rlike whitespace (L1473), six of the 13 patterns are literals that can never match. So the conversion has to fix the patterns, not copy them.

Work

Line numbers refer to CometRegExpJvmSuite unless marked; "parity" means CometRegexParitySuite.

The full per-test add text is in the analysis; this is the condensed version. Use expect_dispatch(rlike) / expect_native(rlike) wherever the Scala test pins the engine. The explain-text checks contains("JVM codegen dispatcher: rlike") map one-to-one onto these. assertCodegenRan is weaker: it passes if anything in the block compiled a kernel.

Existing fixtures to fix and extend

• rlike_java.sql
  • Covers L65-L155.
  • Fix lines 27, 37, 41 and 49.
  • Add '\\d+' and '^(\\w)\\1$' in a projection and in a filter, 'foo(?=bar)', '(?<=foo)bar', '(?<digit>\\d)', '^$', an all-NULL table, and the backreference filter with GROUP BY.
  • All of these patterns are outside the native subset, so pin them with expect_dispatch(rlike).
• rlike_auto_native.sql
  • Covers L224 and L606 (4.0+).
  • Line 28 becomes expect_native(rlike). The file currently says routing is asserted in the Scala suites this issue deletes.
• rlike_rust.sql
  • Covers L501, plus CometExpressionSuite L1473 and L1507.
  • Add expect_native(rlike) for '\\d+' under the opt-in.
  • Add CR, LF and CRLF rows with ^ / $ anchors.
  • Add the curated class, qualifier and anchor set with doubled backslashes.
• routing_regex_enabled.sql
  • Covers L488 and L943.
  • Add expect_dispatch(rlike) for '\\d+', and expect_dispatch(regexp_instr), which no fixture pins today.
• routing_regex_disabled.sql
  • Covers L419 and L549.
  • Add expect_fallback(rlike: spark.comet.exec.scalaUDF.codegen.enabled=false) for the 64-branch pattern and for '\\d+'.
• regexp_extract.sql
  • Covers L682-L722.
  • Fix the backslashes, add rows, and add all-NULL input.
• regexp_extract_all.sql
  • Covers L730-L757.
  • Fix the backslashes, and add '([a-z])(\\d)' with groups 1 and 2, and 'foo(?=ba[rz])'.
• regexp_replace_java.sql
  • Covers L766-L797.
  • Fix the backslashes.
  • Add '(\\w+) (\\w+)' with '$2 $1', 'foo(?=bar)', the empty pattern, and all-NULL input.
• regexp_instr.sql
  • Covers L852-L871.
  • Fix the backslashes, and add groups 1 and 2 and a lookahead.
• split_java.sql
  • Covers L879, L885 and L909.
  • Add 'x,,y', '\\d+' and all-NULL input.

New fixtures

• rlike_java_batches.sql: L169, 5000 generated rows across several batches.
• rlike_routing.sql
  • Covers the analyzer-budget tests: L237-L465 and L531.
  • Write each pattern end to end as expect_dispatch(rlike) or expect_native(rlike).
  • Patterns up to 16,388 characters must be spelled out in full; repeat() would make the pattern non-literal and change the route.
• rlike_native_parity.sql
  • Covers parity L129 and L136.
  • 43 subjects × 30 corpus patterns, in a projection and in a filter, all expect_native(rlike).
  • Write control characters and line terminators as \uXXXX.
• rlike_native_batches.sql: parity L163, with spark.comet.batchSize=64 and 5000 rows.
• rlike_collation.sql and rlike_collation_opt_in.sql (MinSparkVersion: 4.0)
  • L620 and L648 use a collated subject with expect_dispatch(rlike).
  • RLike is not in Spark's collation-coercion list, so the pattern stays a plain literal.
• regexp_extract_batches.sql (L917) and regexp_combined.sql (L929).
• regexp_replace_errors.sql and regexp_replace_errors_row_filter.sql (4.0+)
  • L803 and L828. These are blocked on test: add a Comet SQL test error mode that checks native execution and error-class parity #6617.
  • expect_error(INVALID_REGEXP_REPLACE) can go in now as an interim check.

For the invalid-pattern tests (L144, L272, L579), check the expect_error substring on each profile. Spark's codegen path throws a raw PatternSyntaxException ("Unclosed character class") from generated init code, while the interpreted path wraps it in INVALID_PARAMETER_VALUE.PATTERN.

Blocked

• test: add a Comet SQL test error mode that checks native execution and error-class parity #6617 (error parity): L272, L579, L803, L828.
• test: let Comet SQL tests keep ConstantFolding, exclude optimizer rules and change configs mid-file #6618 (optimizer rules):
  • L531 needs NullPropagation excluded.
  • L634 and L664 need a collated pattern literal, which only exists when ConstantFolding runs.
  • A mid-file statement SET spark.sql.optimizer.excludedRules=... may work today; this was reasoned from the code but not run.
• L440: JDK 21 only, with a 303,103-character pattern. There is no JDK-version directive. L419 covers the same route with a 64-branch pattern, and CometRegexSuite L222 unit-tests the analyzer verdict, so this can stay in Scala.

Stays in Scala

• All 14 CometRegexSuite tests. They are unit tests of CometRegex.supportLevel, the plan-time analyzer, with no Spark session. Several of the patterns don't compile in Java at all.
• L585. It pins a Comet-only failure: the opt-in sends a lookahead to the Rust engine, which rejects it, while Spark succeeds. expect_error needs both engines to fail.

Notes on the Scala tests

• L144 ("invalid pattern falls back to Spark") never checks a fallback. It is a weaker duplicate of L579.
• L943 ("always run via the JVM dispatcher") never pins the dispatcher. Its queries duplicate L682, L730 and L852.
• Covered by other tests: L606 is a subset of L224, and parity L129 is implied by L136.

Done when

• Every test in the table below is accounted for. Either a fixture covers it (the same inputs, configs and assertion strength, or stronger), the fixture that already covers it is cited, or it stays in Scala for the reason given above.
• Each converted Scala test is deleted in the PR that adds its coverage. A suite left empty is deleted and removed from the suite lists in .github/workflows/pr_build_linux.yml and pr_build_macos.yml.
• The new and changed fixtures pass on every Spark profile. Apply run-all-spark-profiles, because pull requests run only the default profile.