[CALCITE-7814] Support the SQL-standard CYCLE clause in recursive CTEs

[vldpyatkov]

Jira Link

CALCITE-7814

Changes Proposed

The implemented CYCLE syntax

WITH RECURSIVE
  edges(src, dst) AS (
    VALUES ('A', 'B'), ('B', 'C'), ('C', 'A')
  ),
  walk(node) AS (
    VALUES ('A')
    UNION ALL
    SELECT e.dst
    FROM walk w JOIN edges e ON e.src = w.node
  )
  CYCLE node SET is_cycle TO 'Y' DEFAULT 'N'
    USING cycle_path
SELECT node, is_cycle, cycle_path
FROM walk;

Explicit mark values must be distinct, non-null Boolean literals or compatible single-character string literals. Omitting TO ... DEFAULT ... uses TRUE and FALSE, respectively.
The clause is represented by SqlCycleClause on SqlWithItem. During validation, CycleRewriter rewrites the query to maintain the cycle mark and an ARRAY<ROW<...>> path. A row that closes a cycle is returned but is not expanded further.

During implementation, I found an issue with comparing ARRAY<ROW<...>> values and fixed the comparer to compare their elements by content. This is needed for correct duplicate elimination when the result includes a cycle path column.

[rubenada]

@vldpyatkov , thanks for this thorough, well-crafted PR. At this point I don't have any further comment, LGTM.
Let's see if we can get any other reviewers opinion on this.

[rubenada]

Taking a final look... I have the impression that the current rewrite approach only works if SqlValidator.Config.identifierExpansion is true (I think the existing tests using CalciteAssert / SqlToRelFixture have this flag set to true, but actually the default value is false).

Having a SELECT * ... test with identifierExpansion=false would be required. IIANM, adding these tests to SqlValidatorTest unveils the issue:

  @Test void testCyclePathWithStarInRecursiveTerm() {
    SqlValidatorTestCase.FIXTURE
        .withValidatorIdentifierExpansion(false)
        .withSql("WITH RECURSIVE t(n) AS (VALUES (1) UNION ALL SELECT * FROM t)\n"
            + "CYCLE n SET c TO 'Y' DEFAULT 'N' USING p\n"
            + "SELECT * FROM t")
        .type("RecordType(INTEGER NOT NULL N, CHAR(1) NOT NULL C, "
            + "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL P) NOT NULL");
  }

  @Test void testCyclePathWithStarInRecursiveTermIdExpansion() {
    SqlValidatorTestCase.FIXTURE
        .withValidatorIdentifierExpansion(true)
        .withSql("WITH RECURSIVE t(n) AS (VALUES (1) UNION ALL SELECT * FROM t)\n"
            + "CYCLE n SET c TO 'Y' DEFAULT 'N' USING p\n"
            + "SELECT * FROM t")
        .type("RecordType(INTEGER NOT NULL N, CHAR(1) NOT NULL C, "
            + "RecordType(INTEGER NOT NULL EXPR$0) NOT NULL ARRAY NOT NULL P) NOT NULL");
  }

The first one fails, but the second one passes.

A potential solution might be forcing the expansion (independently of the flag) in CycleRewriter#prepareQuery (unverified):

private SqlNode prepareQuery(Reference reference) {
    final SqlSelect step = ((SqlCall) queryBody(item.query)).operand(1);
    final SqlNodeList stepSelectList = step.getSelectList();
    final SqlShuttle shuttle = new DeepCopier() {
      @Override public SqlNode visit(SqlNodeList nodeList) {
        return requireNonNull(
            super.visit(nodeList == stepSelectList ? validator.expandStar(stepSelectList, step, false) : nodeList));
      }
      ...

[vldpyatkov]

@rubenada Thanks for catching this! The rewrite now uses the expanded select list saved during validation, even when identifierExpansion is false. I also added tests covering both * and t.* with the flag enabled and disabled.

[rubenada]

Thanks @vldpyatkov . In order to prepare the final merge, could you please squash commits into a single one, keeping Jira number and title as final message: [CALCITE-7814] Support the SQL-standard CYCLE clause in recursive CTEs

If no further comments are received, I'll merge this in the next 48h.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
91.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud