[CALCITE-7782] Large string ARRAY validation is quadratic: 81 GB allocated at 100,000 elements and no result after 20 minutes at 1,000,000

[FrankChen021]

Jira Link

CALCITE-7782

Changes Proposed

Validation of large ARRAY and MAP constructors may adjust many operands to a
common component type. SqlValidatorUtil.adjustTypeForMultisetConstructor
currently calls SqlBasicCall.setOperand separately for every required cast.

Each setOperand invocation creates a new immutable copy of the complete
operand list. When most of n operands require casts, this results in
approximately O(n²) copying.

This change preserves the explicit casts introduced by CALCITE-5948 while
installing the adjusted operands efficiently:

• add SqlCall.setOperandList, with a default implementation that delegates
  to the existing per-operand setter;
• override it in SqlBasicCall to create one immutable operand-list copy;
• collect all required casts in adjustTypeForMultisetConstructor and install
  the resulting list once;
• verify that bulk replacement preserves the existing operand-list snapshot
  behavior.

Other SqlCall implementations retain their existing per-operand behavior
through the default implementation.

Reproducer

Apache Druid exposes the problem after rewriting a large string IN predicate
to a scalar function containing an ARRAY constructor:

EXPLAIN PLAN FOR
SELECT COUNT(*)
FROM foo
WHERE long1 = 8
   OR LOWER(string1) IN ('1', '2', ..., '1000000')

Mixed-width string literals are assigned types such as CHAR(1), CHAR(2),
and CHAR(6), then cast to a common VARCHAR component type. Consequently,
nearly every operand enters the replacement path.

Benchmark

The 81 GB value below is cumulative allocation during one benchmark operation,
not peak heap usage.

The benchmark used Apache Druid commit
d480d66e7fe52ef5daa46eb1af9f33f954be9b7c, JDK 25.0.4.1, JMH 1.37, and
InPlanningBenchmark.queryStringFunctionInSql.

Parameters:

• inSubQueryThreshold = 2147483647
• rowsPerSegment = 500000
• one warmup iteration and three measurement iterations
• GC profiler enabled

String literals	Calcite build	Average time	Allocation
100,000	Official 1.42.0	5,305.626 ms/op	81,622,365,189 B/op
100,000	Patched 1.42.0	1,042.127 ms/op	1,620,149,861 B/op
1,000,000	Official 1.42.0	No completed operation after 20 minutes	Not available
1,000,000	Patched 1.42.0	9,996.863 ms/op	16,187,406,264 B/op

At 100,000 literals, the change is approximately 5.1 times faster and reduces
cumulative allocation by approximately 98%.

The official 1.42.0 million-element operation was stopped after 20 minutes
without producing a JMH score. The patched operation completes in approximately
10 seconds.

Testing

• SqlCallOperandsTest
• ARRAY constructor tests in CalciteSqlOperatorTest
• MAP constructor tests in CalciteSqlOperatorTest
• ARRAY constructor tests in CoreSqlOperatorTest
• MAP constructor tests in CoreSqlOperatorTest
• :core:checkstyleMain
• :core:checkstyleTest
• GitHub Actions suite on the fork PR

All focused tests, style checks, and fork CI passed.

[Copilot]

🟢 Approval recommended

No unresolved blocking issues were identified, and the focused tests and checks passed.

Pull request overview

Optimizes large ARRAY/MAP constructor validation by batching operand replacements and eliminating quadratic immutable-list copying.

Changes:

• Adds bulk operand replacement to SqlCall and SqlBasicCall.
• Applies required casts in one operation.
• Adds operand snapshot behavior tests.

File summaries

File	Description
core/src/test/java/org/apache/calcite/sql/SqlCallOperandsTest.java	Tests replacement and snapshot behavior.
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorUtil.java	Batches constructor operand casts.
core/src/main/java/org/apache/calcite/sql/SqlCall.java	Adds the bulk replacement API.
core/src/main/java/org/apache/calcite/sql/SqlBasicCall.java	Implements efficient bulk replacement.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
56.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[mihaibudiu]

what happens if you use the BasicCall implementation here?

[FrankChen021]

SqlCall cannot use the SqlBasicCall implementation directly because it has no operandList field; other subclasses may store operands differently. The default loop preserves each subclass’s existing setOperand behavior, while SqlBasicCall overrides it to replace its immutable operand list in one copy.

[mihaibudiu]

This looks fine to me, but I still would like an answer to my question before I approve

[FrankChen021]

@mihaibudiu Thank you for reviewing and approving for this. May I know what's the estimated date for next release?

[mihaibudiu]

This is a more appropriate question for the mailing lists.
In fact, the next release has been discussed on the dev mailing list recently
https://lists.apache.org/list.html?dev@calcite.apache.org
The plan is to release a new version of Avatica soon, the RC is supposed to show up in a week or so.
After that a new release of Calcite is planned.
If you are in a hurry, I recommend building against the main branch, without waiting for a release.

[mihaibudiu]

BTW: I think that one of the main obstacles for a faster release cycle is the lack of reviews and of volunteers helping with the release. If you can help with either of these, you can perhaps speed-up the process.

[mihaibudiu]

I plan to merge this PR tomorrow unless there are objections

[mihaibudiu]

BTW: in the future I think a shorter title would be sufficient