Skip to content

allow partial pushdown for semi-scripted predicates - #5565

Merged
Swiddis merged 1 commit into
opensearch-project:mainfrom
Swiddis:fix/like-time-pushdown
Jun 19, 2026
Merged

allow partial pushdown for semi-scripted predicates#5565
Swiddis merged 1 commit into
opensearch-project:mainfrom
Swiddis:fix/like-time-pushdown

Conversation

@Swiddis

@Swiddis Swiddis commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

Description

As like on text returns unsupported operation exceptions instead of predicate analyzer exceptions, this causes the entire expression pushdown to fail since UOE isn't caught in that process. This applies even if some parts of the expression are applicable. So a query like:

source=idx
| where `@timestamp` >= '2023-01-01' and `@timestamp` < '2023-01-04'
| where LIKE(message, '%failed%')
| stats count()

Compiles into a single script that filters documents on a full-table scan. With this fix, it's instead compiled into a script to evaluate the LIKE expression, but the @timestamp predicate is pushed down into DSL:

> echo 'source=logs | where `@timestamp` > "2026-05-22 00:00:00" | where like(raw_log, "nginx") | fields raw_log' | ppl --explain
= Calcite Plan =
== Logical ==
LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])
  LogicalProject(raw_log=[$48])
    LogicalFilter(condition=[ILIKE($48, 'nginx', '\')])
      LogicalFilter(condition=[>($55, TIMESTAMP('2026-05-22 00:00:00':VARCHAR))])
        CalciteLogicalIndexScan(table=[[OpenSearch, logs]])


== Physical ==
CalciteEnumerableIndexScan(table=[[OpenSearch, logs]], PushDownContext=[[PROJECT->[raw_log, @timestamp], SCRIPT->AND(>($1, '2026-05-22 00:00:00'), ILIKE($0, 'nginx', '')), PROJECT->[raw_
  "_source": {
    "includes": [
      "raw_log"
    ]
  },
  "from": 0,
  "query": {
    "bool": {
      "adjust_pure_negative": true,
      "boost": 1.0,
      "must": [
        {
          "range": {
            "@timestamp": {
              "boost": 1.0,
              "format": "date_time",
              "from": "2026-05-22T00:00:00.000Z",
              "include_lower": false,
              "include_upper": true,
              "to": null
            }
          }
        },
        {
          "script": {
            "boost": 1.0,
            "script": {
              "lang": "opensearch_compounded_script",
              "params": {
                "DIGESTS": [
                  "raw_log",
                  "nginx",
                  "\\"
                ],
                "SOURCES": [
                  1,
                  2,
                  2
                ],
                "utcTimestamp": 1781818852859481509
              },
              "source": "{\"langType\":\"calcite\",\"script\":\"rO0ABXQCB3sKICAib3AiOiB7CiAgICAibmFtZSI6ICJJTElLRSIsCiAgICAia2luZCI6ICJMSUtFIiwKICAgICJzeW50YXgiOiAiU1BFQ0lBTCIKICB9LAogICJvcGVyYW5kcyI6IFsKICAgIHsKICAgICAgImR5bmFtaWNQYXJhbSI6IDAsCiAgICAgICJ0eXBlIjogewogICAgICAgICJ0eXBlIjogIlZBUkNIQVIiLAogICAgICAgICJudWxsYWJsZSI6IHRydWUsCiAgICAgICAgInByZWNpc2lvbiI6IC0xCiAgICAgIH0KICAgIH0sCiAgICB7CiAgICAgICJkeW5hbWljUGFyYW0iOiAxLAogICAgICAidHlwZSI6IHsKICAgICAgICAidHlwZSI6ICJWQVJDSEFSIiwKICAgICAgICAibnVsbGFibGUiOiB0cnVlLAogICAgICAgICJwcmVjaXNpb24iOiAtMQogICAgICB9CiAgICB9LAogICAgewogICAgICAiZHluYW1pY1BhcmFtIjogMiwKICAgICAgInR5cGUiOiB7CiAgICAgICAgInR5cGUiOiAiVkFSQ0hBUiIsCiAgICAgICAgIm51bGxhYmxlIjogdHJ1ZSwKICAgICAgICAicHJlY2lzaW9uIjogLTEKICAgICAgfQogICAgfQogIF0KfQ==\"}"
            }
          }
        }
      ]
    }
  },
  "size": 10000,
  "timeout": "1m"
}, requestedTotalSize=10000, pageSize=null, startFrom=0)])

Related Issues

Related to issue 2 identified in #5491

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>
@Swiddis
Swiddis merged commit b2fd268 into opensearch-project:main Jun 19, 2026
36 of 41 checks passed
@Swiddis
Swiddis deleted the fix/like-time-pushdown branch June 19, 2026 22:29
asifabashar pushed a commit to asifabashar/sql that referenced this pull request Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugFix calcite calcite migration releated performance Make it fast! PPL Piped processing language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants