Repository navigation
Conversation
…] fields as parquet
Contributor
Author
|
/cc: @marmbrus |
|
Merged build triggered. |
|
Merged build started. |
Contributor
|
We are incorrectly detecting the |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14807/ |
|
Merged build triggered. |
|
Merged build started. |
Contributor
Author
|
Michael's fix looks good and the test passes now, so unless there's an overhaul of this technique coming soon it probably makes sense to merge this in (pre-1.0). |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
Contributor
|
Merging this. Thanks! |
asfgit
pushed a commit
that referenced
this pull request
May 13, 2014
…tFile() https://issues.apache.org/jira/browse/SPARK-1757 The first test succeeds, but the second test fails with exception: ``` [info] - save and load case class RDD with Nones as parquet *** FAILED *** (14 milliseconds) [info] java.lang.RuntimeException: Unsupported datatype StructType(List()) [info] at scala.sys.package$.error(package.scala:27) [info] at org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetRelation.scala:201) [info] at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$1.apply(ParquetRelation.scala:235) [info] at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$1.apply(ParquetRelation.scala:235) [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) [info] at scala.collection.immutable.List.foreach(List.scala:318) [info] at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) [info] at scala.collection.AbstractTraversable.map(Traversable.scala:105) [info] at org.apache.spark.sql.parquet.ParquetTypesConverter$.convertFromAttributes(ParquetRelation.scala:234) [info] at org.apache.spark.sql.parquet.ParquetTypesConverter$.writeMetaData(ParquetRelation.scala:267) [info] at org.apache.spark.sql.parquet.ParquetRelation$.createEmpty(ParquetRelation.scala:143) [info] at org.apache.spark.sql.parquet.ParquetRelation$.create(ParquetRelation.scala:122) [info] at org.apache.spark.sql.execution.SparkStrategies$ParquetOperations$.apply(SparkStrategies.scala:139) [info] at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) [info] at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) [info] at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) [info] at org.apache.spark.sql.catalyst.planning.QueryPlanner.apply(QueryPlanner.scala:59) [info] at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:264) [info] at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:264) [info] at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:265) [info] at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:265) [info] at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:268) [info] at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:268) [info] at org.apache.spark.sql.SchemaRDDLike$class.saveAsParquetFile(SchemaRDDLike.scala:66) [info] at org.apache.spark.sql.SchemaRDD.saveAsParquetFile(SchemaRDD.scala:98) ``` Author: Andrew Ash <andrew@andrewash.com> Author: Michael Armbrust <michael@databricks.com> Closes #690 from ash211/rdd-parquet-save and squashes the following commits: 747a0b9 [Andrew Ash] Merge pull request #1 from marmbrus/pr/690 54bd00e [Michael Armbrust] Need to put Option first since Option <: Seq. 8f3f281 [Andrew Ash] SPARK-1757 Add failing test for saving SparkSQL Schemas with Option[?] fields as parquet (cherry picked from commit 156df87) Signed-off-by: Reynold Xin <rxin@apache.org>
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
…tFile() https://issues.apache.org/jira/browse/SPARK-1757 The first test succeeds, but the second test fails with exception: ``` [info] - save and load case class RDD with Nones as parquet *** FAILED *** (14 milliseconds) [info] java.lang.RuntimeException: Unsupported datatype StructType(List()) [info] at scala.sys.package$.error(package.scala:27) [info] at org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetRelation.scala:201) [info] at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$1.apply(ParquetRelation.scala:235) [info] at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$1.apply(ParquetRelation.scala:235) [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) [info] at scala.collection.immutable.List.foreach(List.scala:318) [info] at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) [info] at scala.collection.AbstractTraversable.map(Traversable.scala:105) [info] at org.apache.spark.sql.parquet.ParquetTypesConverter$.convertFromAttributes(ParquetRelation.scala:234) [info] at org.apache.spark.sql.parquet.ParquetTypesConverter$.writeMetaData(ParquetRelation.scala:267) [info] at org.apache.spark.sql.parquet.ParquetRelation$.createEmpty(ParquetRelation.scala:143) [info] at org.apache.spark.sql.parquet.ParquetRelation$.create(ParquetRelation.scala:122) [info] at org.apache.spark.sql.execution.SparkStrategies$ParquetOperations$.apply(SparkStrategies.scala:139) [info] at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) [info] at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) [info] at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) [info] at org.apache.spark.sql.catalyst.planning.QueryPlanner.apply(QueryPlanner.scala:59) [info] at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:264) [info] at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:264) [info] at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:265) [info] at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:265) [info] at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:268) [info] at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:268) [info] at org.apache.spark.sql.SchemaRDDLike$class.saveAsParquetFile(SchemaRDDLike.scala:66) [info] at org.apache.spark.sql.SchemaRDD.saveAsParquetFile(SchemaRDD.scala:98) ``` Author: Andrew Ash <andrew@andrewash.com> Author: Michael Armbrust <michael@databricks.com> Closes apache#690 from ash211/rdd-parquet-save and squashes the following commits: 747a0b9 [Andrew Ash] Merge pull request apache#1 from marmbrus/pr/690 54bd00e [Michael Armbrust] Need to put Option first since Option <: Seq. 8f3f281 [Andrew Ash] SPARK-1757 Add failing test for saving SparkSQL Schemas with Option[?] fields as parquet
agirish
pushed a commit
to HPEEzmeral/apache-spark
that referenced
this pull request
May 5, 2022
udaynpusa
pushed a commit
to mapr/spark
that referenced
this pull request
Jan 30, 2024
MaxGekk
added a commit
to MaxGekk/spark
that referenced
this pull request
Oct 9, 2026
Merge apache/spark master so CI can check changes again
MaxGekk
added a commit
to MaxGekk/spark
that referenced
this pull request
Oct 14, 2026
…ark master (apache#806) ### What changes were proposed in this pull request? The CI checkout now builds the branch's own commit. Until now it built the branch merged onto apache/spark's newest master. Every job in `build_and_test.yml` (and the Maven and hosted-Python workflows) checks out its code through one composite action, `.github/actions/checkout-and-sync`, which Varka kept from apache/spark. Outside apache/spark that action: 1. checked out apache/spark at its current master; 2. fetched the fork branch's tip and squash-merged it on top, committing the result as "Merged commit". The action now does this instead, outside apache/spark: - **It checks out `github.sha`**, the commit that triggered the run, from this repository, with full history. There is no merge. Inside apache/spark nothing changes. - **Every job builds the same commit.** The old step fetched the branch tip again in every job, so a push during a run could leave jobs building different commits. - **Change detection keeps its inputs.** `VARKA_DIFF_BASE` and `VARKA_DIFF_HEAD` are still the merge base with Varka's master and the commit under test. `APACHE_SPARK_REF` keeps its name, because `dev/run-tests.py`, `dev/is-changed.py` and two workflow steps read it as the "this is a fork" flag and as a diff base. It now holds that same merge base instead of apache/spark's head. If Varka's master can't be fetched, it falls back to the merge base with apache/spark's master, which runs every module: wasteful, but not wrong. - **The 25 call sites don't change.** The action keeps its inputs and outputs; `ref` is simply unused outside apache/spark. Comments that described the squash merge are updated in `dev/is-changed.py`, `dev/run-tests.py` and `build_and_test.yml`. Two sections of `sql/varka/skills/working-in-this-repo.md` are rewritten for the new behaviour, and `SKILLS.md`'s index is regenerated. ### Why are the changes needed? Merging onto apache/spark's newest master suits a small pull request against apache/spark. It does not suit a fork that differs from apache/spark by thousands of commits. Every run re-merged all of Varka onto whatever apache/spark had merged that hour, which caused three problems: - **One upstream edit to a file Varka rewrote breaks every open branch at once.** At 04:33 CEST on 14 October, SPARK-60210 added a badge to `README.md`. Every fork run since then fails at "Check changes" with a merge conflict, before anything builds; apache#789's and apache#805's runs did. The same thing happened with `build_and_test.yml` on 9 October (apache#690). Each time, a sync pull request has to land, and then every open branch has to merge it and run CI again. - **No run ever built what Varka's master actually contains.** A branch could go red because of an upstream change Varka didn't have. One upstream test failed on every branch for a week while it existed nowhere in this repository. - **Change detection had to work around the merge.** VARKA-106 computed the merge base before the squash threw away the history. That workaround becomes the plain behaviour. After this change, apache/spark enters Varka only through the sync pull requests that merge apache/spark's master into Varka's master. An upstream conflict or test failure then shows once, in that pull request's own run, instead of on every open branch. The trade-off: a branch no longer warns early that it would conflict with apache/spark. Varka sends nothing upstream as-is, so it doesn't need that warning. ### Does this PR introduce _any_ user-facing change? No. CI only. ### How was this patch tested? - **The fork CI run on this branch.** Its base is Varka's master without apache#804's README fix, so getting past "Check changes" shows that no merge onto apache/spark happens any more. The precondition's "changed files vs base" line should show this pull request's six files. - **Local checks:** the action file parses as YAML, `dev/is-changed.py` and `dev/run-tests.py` compile and pass `ruff check` and `ruff format`, `dev/varka_toc.py --check` passes on `SKILLS.md`, and the pre-commit hook reports no findings. `actionlint` and `shellcheck` are not installed on this machine. apache#804, the sync pull request that fixes today's conflict, stays useful after this merges: it is the regular sync, it just stops being urgent. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/SPARK-1757
The first test succeeds, but the second test fails with exception: