This repository was archived by the owner on Dec 15, 2021. It is now read-only.
Enable user to specify write options, hinting about issues with nested records#59
Merged
Merged
Conversation
added 3 commits
January 24, 2018 18:55
When writing nested fields to BigQuery one has to specify a namespace prefix for Avro - otherwise nested fields will have a namespace starting with a dot which the BigQuery Avro read will not accept.
To load schemas with nested fields, one has to specify an Avro Namespace.
jobegrabber
changed the title
lverzosa
added a commit
to lverzosa/spark-bigquery
that referenced
this pull request
Apr 26, 2019
…th nested records (spotify#59)" This reverts commit 7f65455.
lverzosa
added a commit
to lverzosa/spark-bigquery
that referenced
this pull request
May 1, 2019
* Updating versions * Revert "Update README.md" This reverts commit 55c75ec. * Revert "Added support for Spark 2.4.0 and Scala 2.12. Adjusted code slightly since Databricks Avro library moved into Spark with small syntax changes. (spotify#69)" This reverts commit ee9136b. * Revert "Update README.md (spotify#68)" This reverts commit a2d5a63. * Revert "Create daily partitioned table when necessary (spotify#67)" This reverts commit f5b4815. * Revert "Add maintenance mode notice" This reverts commit ab8fd9c. * Revert "Enable user to specify write options, hinting about issues with nested records (spotify#59)" This reverts commit 7f65455. * Revert "Configuration parameter for intermediate dataset id (spotify#63)" This reverts commit a0ecb3f. * Revert "Make a defensive copy to stop polution of sparkconf (spotify#60)" This reverts commit ffcd9a6. * Revert "Support setting query job priority (spotify#54)" This reverts commit 07e012a. * Fixing GoogleCredentials to use supplied file
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writing nested records to BigQuery with the current implementation was not possible: BigQuery is seemingly unable to load Avro Namespaces with a leading dot (
.nestedColumn) on nested records.spark-bigquery utilizes temporary Avro files written to Cloud Storage.
This PR enables users to specify write options such as setting the Avro Namespace which will enable users to write nested records to BigQuery.
Additionally, this PR adds a hint and link to how BigQuery handles the import of Avro files (for example, arrays of arrays are not supported).