Repository navigation
Conversation
…itter (delta-io#5337) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/5337/files) to review incremental changes. - [**stack/kernel_uc_committer_max_published_delta_version**](delta-io#5337) [[Files changed](https://github.com/delta-io/delta/pull/5337/files)] --------- #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [X] Kernel - [ ] Other (fill in here) ## Description This PR adds the `maxKnownPublishedDeltaVersion` to the `CommitMetadata` API. It then updates the `UC Commiter` to use this value and send it to UC server. UC Server will then use this value to clean up its state. ## How was this patch tested? New E2E test. ## Does this PR introduce _any_ user-facing changes? No.
…ta-io#5352) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description It fixes a problem with renaming columns with `dataSkipping` enabled. Currently the following column rename will fail ```sql create or replace table main.default.delta_rename_test (c int) tblproperties ("delta.columnMapping.mode" = "name", "delta.dataSkippingStatsColumns" = "c"); alter table main.default.delta_rename_test rename column c to `c#2`; ``` This PR addresses the issue. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Tests were added to [DeltaColumnRenameSuite.scala](spark/src/test/scala/org/apache/spark/sql/delta/DeltaColumnRenameSuite.scala) <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No <!-- Please read [go/dbr-dev-guides](https://go/dbr-dev-guides/) before creating a pull request against files shared between Databricks Runtime and open-source Apache Spark. For example, this includes files under the `org.apache.spark` package path. If in doubt, please reach out to a member of the open-source team for help. -->
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [X] Kernel - [ ] Other (fill in here) ## Description This PR extends the Delta Kernel API to support creating collated predicates for partition pruning when the provided predicate includes a collated component. In delta-io#4250, we already merged the logic so that the default engine fails when trying to evaluate a non-default collated predicate. Therefore, the remaining work for partitions is only on the API side. ## How was this patch tested? New tests. ## Does this PR introduce _any_ user-facing changes? No.
…ta-io#5340) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/5340/files) to review incremental changes. - [**stack/kernel_snapshot_crc_statistics_and_write**](delta-io#5340) [[Files changed](https://github.com/delta-io/delta/pull/5340/files)] --------- #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [X] Kernel - [ ] Other (fill in here) ## Description This PR adds: - `Snapshot::getStatistics` and `SnapshotStatistics` APIs - `ChecksumWriteMode` API - `Snapshot::writeChecksum(Engine, ChecksumWriteMode)` API - Updates tests This PR enables the e2e flow of: - connector performs a write - connector asks for the post commit snapshot - connector can look at the statistics and see what is the available checksum write mode - for a post-commit snapshot it will be either SIMPLE or FULL (no one else should have written the crc yet) - the connector can decide if it wants to pay the cost of the SIMPLE or FULL crc write, and then invoke `Snapshot::writeChecksum` with the mode they are willing to pay ## How was this patch tested? New UTs. Updated existing write-with-crc-suites to run using this new logic, too. ## Does this PR introduce _any_ user-facing changes? New Snapshot APIs.
…5142) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (Docs) ## Description This PR introduces a new Github Actions workflow which will automatically build and deploy the docs site and any desired version(s) of API docs. This workflow runs only under one of the following conditions: 1. Whenever a PR is merged into `master` which includes changes the `/docs` directory (most common case). 2. If there is a new [release](https://github.com/delta-io/delta/releases) (less common — mainly used for API doc updates). 3. Manually (useful for debugging purposes; only users with the `collaborators` permission to this repo can run it). Here's an example of a working deployment: - [Github Actions log](https://github.com/jakebellacera/delta/actions/runs/17314645500) - [Test deployment](https://vocal-bonbon-3bb61a.netlify.app/) ### How it works 1. When one of the conditions above is met, the `publish_docs` workflow is kicked off. 2. Builds one or more version(s) of the Delta Lake API docs. 3. Builds the docs site with the specified Delta Lake API docs. 5. Uploads the site to Netlify. ### Specifying API docs versions This workflow can build multiple versions of Delta Lake APIs. To do so, just specify the version(s) in the `publish_docs` workflow. Here's what's currently defined: https://github.com/delta-io/delta/blob/89d77c7a94f274ece59a9a95c56edf5a95df6bfa/.github/workflows/publish_docs.yaml#L14-L29 The workflow will check out the `ref` of this repo and use it to generate the API docs. The `name` is used to specify the human-readable folder to the API docs. For example, "latest" is using the ref [v4.0.0](https://github.com/delta-io/delta/tree/v4.0.0), while "3.3.2" is using the ref [v3.3.2](https://github.com/delta-io/delta/tree/v3.3.2). This allows for more flexibility. ### Additional changes required before merging Add the following [variables](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-variables) to this Github repo: - `NETLIFY_SITE_ID` - [Netlify Site ID](https://docs.netlify.com/api-and-cli-guides/cli-guides/get-started-with-cli/) we want to deploy to Add the following [secrets](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets) to this Github repo: - `NETLIFY_AUTH_TOKEN` - [Netlify user PAT](https://docs.netlify.com/api-and-cli-guides/cli-guides/get-started-with-cli/#obtain-a-token-in-the-netlify-ui) to authenticate Netlify CLI with ### Other changes - Updated the v3 API docs to v3.3.2 - Upgraded Node.js dependencies - Updated the scala version for building v4 API docs (note: due to our automation building an older ref of Delta Lake v4, this won't apply until the next v4 release). --------- Co-authored-by: Robert Pack <42610831+roeap@users.noreply.github.com>
## Description I'm seeing random failures in our CI jobs for python tests in `test_deltatable` (I would say 1 in 100 or 1000 executions), falling in two categories: 1. Creating a table in tests fails with [DELTA_CREATE_TABLE_WITH_NON_EMPTY_LOCATION] 2. `shutil.rmtree` called to delete the temp dir in `tearDown` after each tests randomly due to missing `.tmp` files Many tests are using a static table name, which breaks test isolation and explains 1.: if a test incorrectly cleans up the table, then a following test will fail to recreate it. I wasn't able to exactly root cause 2., but it is easily fixed by ignoring errors during tearDown, which is the reasonable thing to do. ## How was this patch tested? I didn't manage to reproduce the flakiness locally or on CI to validate these fixes, but I'm reasonably convinced this will remove or at least reduce the flakiness.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description Upgrade Iceberg dependency to 1.9.0 ## How was this patch tested? Existing Tests
## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/5371/files) to review incremental changes. - [**stack/kernel_metrics_report_to_json**](delta-io#5371) [[Files changed](https://github.com/delta-io/delta/pull/5371/files)] --------- #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description This PR adds a new `toJson` method to `MetricsReport`. For example, this will allow the `LoggingMetricsReporter` in the kernel default engine to Log _any_ MetricsReport (e.g. those emitted by a Committer) and not only those defined within Kernel-API. ## How was this patch tested? Updated UTs. ## Does this PR introduce _any_ user-facing changes? No.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [X] Kernel - [ ] Other (fill in here) ## Description Based off of delta-io#5305. Review the latest commit [here](delta-io@c406b93) for the changes in this PR. Adds support for `loadCommitRange` to the `UCCatalogManagedClient`. ## How was this patch tested? Unit tests. ## Does this PR introduce _any_ user-facing changes? Yes, new API to load CommitRange.
#### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Fixed the expected ordering of entries in a merge test with identity columns. The old order was dependent on the order merge writes rows into the table, which is not deterministic. ## How was this patch tested? Unit test passes. ## Does this PR introduce _any_ user-facing changes? No
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (Delta Sharing) ## Description When there is a version defined in the query request, refresh token isn't allowed. - Added a useRefreshToken parameter to DeltaSharingUtils.getRefresherForGetFiles() to control whether refresh tokens should be used when refreshing pre-signed URLs - Modified DeltaFormatSharingSource to set useRefreshToken = false for streaming queries, ensuring refresh tokens are always ignored for version-specific snapshot queries ## How was this patch tested? New test case ## Does this PR introduce _any_ user-facing changes? No --------- Co-authored-by: Sun Cao <sun.cao+data@databricks.com>
…5360) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description In this PR, the `constructDataSkippingFilter` method is extended to support creating `DataSkippingPredicate`s for collated data skipping. The collation stats schema used for skipping is [in this document](https://docs.google.com/document/d/1cwztlKt7b2hWF6Uu1S895ko6jPfRlP9x-V5POUcXtXk/edit?tab=t.0#heading=h.gapzrbgs8fs7). This change only updates the construction logic of `DataSkippingPredicate` to make it collation-aware. The Kernel does not yet read collated statistics - that functionality will be introduced in a follow-up PR ([here](https://github.com/delta-io/delta/blob/master/kernel/kernel-api/src/main/java/io/delta/kernel/internal/ScanImpl.java#L368) we will extend schema to include collation stats schema). Additionally, E2E tests for default engine to fail evaluating these predicates will be added in follow-up. ## How was this patch tested? New tests. ## Does this PR introduce _any_ user-facing changes? No.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (README) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Update readme to specify Java 11 for sbt build, which is needed after Iceberg dependency upgrade. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Verified locally. With Java 1.8, sbt compilation fails. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No, README update for developers.
…npublished versions (delta-io#5372) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/5372/files) to review incremental changes. - [**stack/kernel_uc_catalog_managed_crc_log_segment_construction**](delta-io#5372) [[Files changed](https://github.com/delta-io/delta/pull/5372/files)] --------- #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [X] Kernel - [ ] Other (fill in here) ## Description This PR adds two new tests (1 is a unit test in Kernel-API, the other is a mocked E2E read+write in delta-unity JAR) that ensures that Kernel can correctly construct a LogSegment using CRC files for versions that do not have accompanying published deltas. ## How was this patch tested? New tests only. ## Does this PR introduce _any_ user-facing changes? No.
…eckpoint Versions (delta-io#5355) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This PR updates the Delta Protocol specification to clarify that JSON commit files at checkpoint versions must be preserved during metadata cleanup. This ensures that commitInfo actions remain accessible for table features such as In-Commit Timestamps (ICT). The current Delta Protocol's metadata cleanup section could be interpreted to allow deletion of JSON commit files when a checkpoint exists at the same version. However, this creates a critical problem for ICT-enabled tables: 1. Checkpoints don't preserve commitInfo actions - The protocol explicitly states that checkpoints must not preserve commit provenance information 2. ICT writers need previous commit timestamps - ICT-enabled tables need to read the previous commit's inCommitTimestamp to ensure monotonicity Thus, if a checkpoint exists without its corresponding JSON commit file, the commitInfo is lost, causing writes to the table to fail. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…elta-io#5375) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description Switch Iceberg dependency to shaded Iceberg-core in clone logics ## How was this patch tested? Existing UTs
…-io#5388) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Adds a `Utils.flatMap` utility method for flattening nested `CloseableIterator<CloseableIterator<T>>` structures into `CloseableIterator<T>`. It will be helpful for use case like delta-io#5290 ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Unit tests ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
…rgeInto (delta-io#5390) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR extracts the inline `getActions` helper function into a standalone private method `getLeafActionsForSchema` in `ResolveDeltaMergeInto`. This is the first step in a series of refactorings to improve code organization and readability. The `getLeafActionsForSchema` function generates `DeltaMergeAction` sequences for every leaf field in a schema, recursively unfolding nested struct columns. The change is needed for fixing struct null expansion issues in MERGE. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Existing tests. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Fix an issue observed in https://github.com/delta-io/delta/pull/5320/files#r2456513386. The changes is due to api difference in different spark version, so introduce a shim to take care of that ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Existing test with a new workflow setup ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
…ng stats" test (delta-io#5386) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This PR disables a test if the feature flag for the fix is disabled. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Test-only fix ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…ta-io#5396) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This PR extracts the star resolution logic (`UPDATE SET *` and `INSERT *`) into a new standalone private method `resolveStar` in `ResolveDeltaMergeInto`. This is required for the struct null expansion fix in MERGE. ## How was this patch tested? Existing tests. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
delta-io#5401) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Fix an issue observed in https://github.com/delta-io/delta/pull/5320/files#r2456513386. The changes is due to api difference in different spark version and delta-io#5389 does not quite fix it, this pr fixes that ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Existing test with a new workflow setup ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
…ta skipping (delta-io#5398) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Currently, partition-like data skipping supports partial rewrites of conjunctions when rewriting a data filter as a partition-like data skipping predicate. For example, we currently might rewrite: `(UPPER(str_clustering_col) = 'ABC' AND UPPER(non_clustering_col) = 'DEF') OR clustering_col > 5` to `(UPPER(minValues.str_clustering_col) = 'ABC' OR minValues.clustering_col > 5) AND (... stats validation ...)`. The assumption that both sides of a conjunction may be safely rewritten held for traditional data skipping, where we explicitly handled recursion, ensuring that all of the expressions above the current filter are conjunctive (making it safe to return false positives, but not false negatives). However, this assumption no longer holds for partition-like skipping: the parent expressions of an AND expression might not necessarily be conjunctive, making false positives by partial rewrites no longer acceptable. For example, consider: `NOT(UPPER(str_clustering_col) = 'ABC' AND UPPER(non_clulstering_col) = 'DEF'))`. This expression currently would be rewritten to `NOT(UPPER(minValues.str_clustering_col) = 'ABC')) AND (...stats validation...)`, which is invalid - because of the `NOT`, we can no longer tolerate false positives from the `AND` expression. As a result, this PR removes: 1. The partition-like skipping logic allowing partial rewrites of each branch of an `AND` expression, as this logic is unsafe (as noted above). 2. The partition-like skipping logic that pushes down `NOT` through `OR` to get an `AND` expression: this logic is no longer needed (we don't actually have any preference for `AND` over `OR` anymore). This change will reduce the coverage of partition-like data skipping, but must be made to ensure correctness. In theory, it's possible to increase partition-like coverage by tracking the parent expressions of the current expression during the partition-like rewrite, applying partial rewrites for `AND` expressions for a known-good set of parent expressions, but this would be a significant change that likely would not yield a significant increase in partition-like skipping coverage. ## How was this patch tested? See test changes. ## Does this PR introduce _any_ user-facing changes? No.
…elta-io#5405) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR extracts the merge clause resolution logic into a new standalone private method `resolveClause` in `ResolveDeltaMergeInto`. This is required for the struct null expansion fix in MERGE. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Existing tests. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No.
#### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Split the slow base traits "MergeIntoLowShuffleMiscTests", "MergeIntoNestedStructEvolutionTests", and "MergeIntoSchemaEvolutionBaseTests" into two, and regenerated the tests accordingly. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> N/A ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> N/A
…heckpoint manifest (delta-io#5406) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description As titled. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Through existing UTs. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…sses (delta-io#5394) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> As part of kernel integration effort we observed some difficulties in pattern matching e.g. delta-io#5369 this PR mitigates it by introducing some finer grain error required in the issue ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Unit test ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description In this PR, the `isCollatedSkipping` parameter is removed from methods like `isSkippingEligibleDataType`. The idea behind this change is that the parameter only adds noise, since collated skipping is just one specific case of skipping. ## How was this patch tested? Existing tests. ## Does this PR introduce _any_ user-facing changes? No.
…DeltaMergeInto (delta-io#5412) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR extracts computePostEvolutionTargetSchema from ResolveDeltaMergeInto. The change is required for struct null expansion fix in MERGE. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Existing tests. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description If we have a table with an integer column `file_name` and run a query like: ```sql select * from file_metadata_test where _metadata.file_name = 'part-00000-fab85aae-5302-49c9-8e4d-747da36e5ae9.c000.zstd.parquet' ``` we get an error: ``` [[CAST_INVALID_INPUT](https://docs.databricks.com/error-messages/error-classes.html#cast_invalid_input)] The value 'part-00000-fab85aae-5302-49c9-8e4d-747da36e5ae9.c000.zstd.parquet' of the type "STRING" cannot be cast to "BIGINT" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. SQLSTATE: 22018 ``` This happens because `FileSourceStrategy#rebindFileSourceMetadataAttributesInFilters` flattens `_metadata` struct, leaving just `file_name`, and then `DataSkippingReader` tries to use min-max stats of the actual `file_name` column to prune files and fails when it tries to cast the right-hand side of the equality predicate. ## What changes were proposed in this pull request? Fix the bug by not using `dataFilters` involving `_metadata` fields for data skipping. ## How was this patch tested? I added a new test case to `DataSkippingDeltaTests.scala`. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> Signed-off-by: Uros Stojkovic <uros.stojkovic@databricks.com>
…e api (delta-io#5290) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This PR introduces api to return per commit action iterator in kernel get change api, it will help to implementing current structure streaming logic using kernel Case 1: need to find commit boundary and inject offset <img width="2252" height="1476" alt="problem 1" src="https://github.com/user-attachments/assets/8c5d2324-af05-4ca6-8816-ba24b7f401bd" /> Case 2: skip commits without read through iterator: no need to iterate all action in 2.json and directly go to 3.json <img width="2252" height="1476" alt="problem 2" src="https://github.com/user-attachments/assets/4032bcc8-d948-48a1-8e49-9ad16e909a0c" /> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Unit ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
How was this patch tested?
Does this PR introduce any user-facing changes?