Skip to content

[VL] Support empty dataframe write - #2435

Merged
JkSelf merged 3 commits into
apache:mainfrom
ulysses-you:empty-parquet-write
Jul 25, 2023
Merged

JkSelf merged 3 commits into
apache:mainfrom
ulysses-you:empty-parquet-write

Conversation

@ulysses-you

@ulysses-you ulysses-you commented Jul 21, 2023 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

If the write data is empty, we should write a valid parquet file rather than an empty file.

val df = spark.emptyDataFrame.select(lit(1).as("i"))
df.write.format("velox").save("test3")

before:
image

after:
image

How was this patch tested?

Add test

@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@ulysses-you ulysses-you changed the title [vl] Support empty parquet write [VL] Support empty parquet write Jul 21, 2023
@ulysses-you
ulysses-you force-pushed the empty-parquet-write branch from a2d615c to 1840565 Compare July 22, 2023 05:17
@ulysses-you
ulysses-you force-pushed the empty-parquet-write branch from 1840565 to d71162e Compare July 25, 2023 02:41
@ulysses-you ulysses-you changed the title [VL] Support empty parquet write [VL] Support empty dataframe write Jul 25, 2023
Comment thread ep/build-velox/src/get_velox.sh Outdated

VELOX_REPO=https://github.com/oap-project/velox.git
VELOX_BRANCH=main
VELOX_REPO=https://github.com/NetEase/gluten-velox

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please help to revert this change. Thanks.

@ulysses-you
ulysses-you requested a review from JkSelf July 25, 2023 07:14
@JkSelf
JkSelf merged commit 6b93602 into apache:main Jul 25, 2023
@ulysses-you
ulysses-you deleted the empty-parquet-write branch July 25, 2023 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants