[CALCITE-5948] Use explicit casting if element type in ARRAY/MAP does not equal derived component type by taoran92 · Pull Request #3395 · apache/calcite

taoran92 · 2023-08-27T07:07:34Z

https://issues.apache.org/jira/browse/CALCITE-5948

tanclary · 2023-08-31T15:34:52Z

+      } else {
+        elementType = oddType;
+      }
+      if (operandTypes.get(i).equalsSansFieldNames(elementType)) {


can this be rewritten as if(!operandTypes.get(i)....) { call.setOperand() }

yes, good point. fixed. thanks tanner for reviewing.

taoran92 · 2023-09-04T16:19:24Z

@tanclary hi, tanner. I have solved some comments. If you have time, could you help to review it again? thanks!

taoran92 · 2023-09-06T12:38:46Z

@tanclary hi, Tanner. Sorry to ping you. It's been two weeks, could you help to review it again? Looking forward your comments.

JiajunBernoulli · 2023-09-18T12:55:37Z

+------------------+
+| EXPR$0           |
+------------------+
+| [a  , null, bcd] |


Why is there an extra space?

@JiajunBernoulli thanks for reviewing. this is a import change in this PR.

if we have array('A', 'AB'), the derived component type is char(2), the old behavior not cast 'A' to char(2) in runtime to match derived component type. The correct behavior let 'A' match the char(2), 'A' cast to char(2), then it will fill with extra spaces by default in calcite.

This is calcite and sql standard limitation. Of course, this does look weird, actually calcite has other ways to remove spaces. We can activate SqlConformanceEnum.PRAGMATIC_2003 to let shouldConvertRaggedUnionTypesToVarying be true, it will change array/map char type to varchar type to skip these spaces.

More details can be found in javadoc https://github.com/apache/calcite/blob/d9dd3ac8a9f695e111a0a5e77f45b61b90f4b5b6/core/src/main/java/org/apache/calcite/sql/validate/SqlConformance.java#L465C7-L465C7

I also have added some cases in the PR to show this difference.

@JiajunBernoulli hi, Jiajun. If you have time, could you help to review it again? thanks

Ok, Thank you for your clear explanation.

tanclary

Hi @chucheng92 Sorry for delay, thanks for addressing the changes. I'm okay to merge this as soon as @JiajunBernoulli is too.

taoran92 · 2023-09-20T16:02:14Z

Thanks Tanner and Jiajun for reviewing. All comments are resolved/confirmed.
I have rebased main and squashed the commits and made commit name to match Jira name.
hi @tanclary If you have time, could you help to merge this?

taoran92 · 2023-09-21T00:36:22Z

@julianhyde hi, julian. I noticed that you gave a comment in the ticket to suggest giving a test case for calcite-5960. I have added it, if you have time, could you help to review it? If you approve that, then i will squash the commits. thanks!

taoran92 · 2023-09-21T08:50:19Z

@julianhyde hi, julian. I noticed that you gave a comment in the ticket to suggest giving a test case for calcite-5960. I have added it, if you have time, could you help to review it? If you approve that, then i will squash the commits. thanks!

calcite-5960 has been resolved.

taoran92 · 2023-09-21T15:29:26Z

thanks Julian and Vladimir for reviewing the related commit calcite-5960. Considering that commit has been resolved, so I have rebased this PR. Currently all comments are resolved.

Hi, @tanclary sorry to ping you, if you have time, could you help to recheck/merge this?
thank you all again for patient reviewing.

… not equal derived component type

sonarqubecloud · 2023-09-25T14:27:57Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
15 Code Smells

100.0% Coverage
5.2% Duplication

taoran92 · 2023-09-27T17:10:52Z

hi Tanner @tanclary , if you are free, could you help to take a look again? Sorry to bother you.

taoran92 · 2023-10-06T02:37:17Z

@tanclary Hi, Tanner, I have updated the commit & jira name and added javadocs. If you have time, could you help to review it again? thank you.

tanclary

Hi @chucheng92 sorry for the delay, I was out unexpectedly for a couple of weeks. Thank you for resolving all of the comments, I think this looks great.

taoran92 · 2023-10-09T16:21:02Z

-    // Result should be "EXPR$0=[1, 1.1]\n"; [CALCITE-4850] logged.
    CalciteAssert.that()
        .query("select array[1, 1.1]")
-        .returns("EXPR$0=[0E+1, 1.1]\n");


@tanclary thanks for patient reviewing! btw, I think we can safely close CALCITE-4850. The result has been corrected.

FrankChen021 · 2026-09-14T13:52:29Z

This change appears to introduce a severe performance regression when validating very large array constructors containing mixed-width string literals.

We reproduced it using Apache Druid’s InPlanningBenchmark.queryStringFunctionInSql:

EXPLAIN PLAN FOR
SELECT COUNT(*)
FROM foo
WHERE long1 = 8
   OR LOWER(string1) IN ('1', '2', ..., '1000000')

Benchmark parameters:

inClauseLiteralsCount=1000000
inSubQueryThreshold=2147483647
rowsPerSegment=500000

Results:

Calcite version	Result
1.35.0	10.39 s/op, 14.94 GB allocated/op
1.37.0	No completed operation after 20 minutes
1.41.0	No completed operation after 20 minutes
1.42.0	No completed operation after 20 minutes

The regression first appears after upgrading from Calcite 1.35 to 1.37; this PR was included in Calcite 1.36.

A JFR profile of the 100,000-string case attributed most allocation pressure to:

ImmutableList.copyOf: 50.29%
Platform.copy: 41.28%

The dominant stack was:

SqlArrayValueConstructor.inferReturnType
  -> SqlValidatorUtil.adjustTypeForArrayConstructor
  -> SqlValidatorUtil.adjustTypeForMultisetConstructor
  -> SqlBasicCall.setOperand
  -> ImmutableNullableList.copyOf

adjustTypeForMultisetConstructor invokes setOperand separately for every element requiring a cast. SqlBasicCall.setOperand copies the complete immutable operand list each time. For large mixed-width string arrays, this produces approximately quadratic copying and extreme allocation pressure.

Numeric arrays generally do not reproduce the same degradation because their elements commonly already have the derived component type.

Related Druid investigation: apache/druid#20326 (comment)

mihaibudiu · 2026-09-14T16:58:02Z

Commenting on a closed PR is not useful, you should open a new issue in JIRA.

taoran92 force-pushed the CALCITE-5948 branch 4 times, most recently from 1556351 to 8075f6c Compare August 29, 2023 03:37

tanclary requested changes Aug 31, 2023

View reviewed changes

taoran92 requested a review from tanclary September 4, 2023 16:19

tanclary force-pushed the main branch from 4804912 to 00db001 Compare September 6, 2023 00:42

taoran92 force-pushed the CALCITE-5948 branch from a48da42 to 824f2bb Compare September 6, 2023 13:43

JiajunBernoulli reviewed Sep 18, 2023

View reviewed changes

tanclary approved these changes Sep 18, 2023

View reviewed changes

taoran92 force-pushed the CALCITE-5948 branch from 824f2bb to 57737be Compare September 20, 2023 13:35

taoran92 changed the title ~~[CALCITE-5948] Explicit casting should be made if the type of an element in ARRAY/MAP not equals with the derived component type~~ [CALCITE-5948] Explicit casting should be made if the type of an element in ARRAY/MAP not equals the derived component type Sep 20, 2023

taoran92 force-pushed the CALCITE-5948 branch from 57737be to 06464ea Compare September 20, 2023 14:33

taoran92 requested a review from tanclary September 20, 2023 16:02

taoran92 force-pushed the CALCITE-5948 branch from 95ec19f to a30469b Compare September 21, 2023 00:41

vlsi mentioned this pull request Sep 21, 2023

Bump Calcite: 1.26 -> 1.32 vlsi/mat-calcite-plugin#30

Closed

vlsi reviewed Sep 21, 2023

View reviewed changes

Comment thread core/src/main/java/org/apache/calcite/adapter/enumerable/RexToLixTranslator.java Outdated

taoran92 force-pushed the CALCITE-5948 branch 2 times, most recently from 8704704 to 4feaa6b Compare September 21, 2023 08:41

tanclary requested changes Sep 21, 2023

View reviewed changes

Comment thread babel/src/test/resources/sql/big-query.iq

Comment thread core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorUtil.java

Comment thread core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorUtil.java

taoran92 changed the title ~~[CALCITE-5948] Explicit casting should be made if the type of an element in ARRAY/MAP not equals the derived component type~~ [CALCITE-5948] Use explicit casting if element type in ARRAY/MAP does not equal derived component type Sep 22, 2023

taoran92 force-pushed the CALCITE-5948 branch 2 times, most recently from a9befa2 to 90de969 Compare September 22, 2023 09:58

taoran92 requested a review from tanclary September 25, 2023 13:09

[CALCITE-5948] Use explicit casting if element type in ARRAY/MAP does…

f64e4ac

… not equal derived component type

taoran92 force-pushed the CALCITE-5948 branch from 90de969 to f64e4ac Compare September 25, 2023 14:00

tanclary approved these changes Oct 9, 2023

View reviewed changes

tanclary merged commit 342cf60 into apache:main Oct 9, 2023

taoran92 commented Oct 9, 2023

View reviewed changes

FrankChen021 mentioned this pull request Sep 14, 2026

Large numeric IN predicates cause slow SQL planning apache/druid#20326

Open

Conversation

taoran92 commented Aug 27, 2023

Uh oh!

tanclary Aug 31, 2023

Choose a reason for hiding this comment

Uh oh!

taoran92 Sep 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

taoran92 commented Sep 4, 2023

Uh oh!

taoran92 commented Sep 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JiajunBernoulli Sep 18, 2023

Choose a reason for hiding this comment

Uh oh!

taoran92 Sep 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taoran92 Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JiajunBernoulli Sep 20, 2023

Choose a reason for hiding this comment

Uh oh!

tanclary left a comment

Choose a reason for hiding this comment

Uh oh!

taoran92 commented Sep 20, 2023

Uh oh!

taoran92 commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

taoran92 commented Sep 21, 2023

Uh oh!

taoran92 commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud Bot commented Sep 25, 2023

Uh oh!

taoran92 commented Sep 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taoran92 commented Oct 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tanclary left a comment

Choose a reason for hiding this comment

Uh oh!

taoran92 Oct 9, 2023

Choose a reason for hiding this comment

Uh oh!

FrankChen021 commented Sep 14, 2026

Uh oh!

mihaibudiu commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

taoran92 Sep 1, 2023 •

edited

Loading

taoran92 commented Sep 6, 2023 •

edited

Loading

taoran92 Sep 18, 2023 •

edited

Loading

taoran92 Sep 19, 2023 •

edited

Loading

taoran92 commented Sep 21, 2023 •

edited

Loading

taoran92 commented Sep 21, 2023 •

edited

Loading

taoran92 commented Sep 27, 2023 •

edited

Loading

taoran92 commented Oct 6, 2023 •

edited

Loading