[SPARK-6888][SQL] Make the jdbc driver handling user-definable by rtreffer · Pull Request #5555 · apache/spark

rtreffer · 2015-04-17T10:46:06Z

Replace the DriverQuirks with JdbcDialect(s) (and MySQLDialect/PostgresDialect)
and allow developers to change the dialects on the fly (for new JDBCRRDs only).

Some types (like an unsigned 64bit number) can be trivially mapped to java.
The status quo is that the RRD will fail to load.
This patch makes it possible to overwrite the type mapping to read e.g.
64Bit numbers as strings and handle them afterwards in software.

JDBCSuite has an example that maps all types to String, which should always
work (at the cost of extra code afterwards).

As a side effect it should now be possible to develop simple dialects
out-of-tree and even with spark-shell.

AmplabJenkins · 2015-04-17T10:47:10Z

Can one of the admins verify this patch?

marmbrus · 2015-04-17T18:17:45Z

ok to test

SparkQA · 2015-04-17T18:23:31Z

Test build #30492 has started for PR 5555 at commit 3e98f74.

SparkQA · 2015-04-17T18:25:03Z

Test build #30492 has finished for PR 5555 at commit 3e98f74.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-17T18:25:04Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30492/
Test FAILed.

marmbrus · 2015-04-21T22:10:22Z

Thanks for working on this! Looks like there are some lines that need to be wrapped:

[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala:64: File line length exceeds 100 characters
[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala:138: File line length exceeds 100 characters
[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala:166: File line length exceeds 100 characters
[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala:191: File line length exceeds 100 characters

rtreffer · 2015-04-24T20:40:52Z

Thank you for the fast feedback, I'm quite impressed by the feedback speed/quality on the spark project.

I was traveling last week, so the fix was a bit delayed.

rtreffer · 2015-04-25T13:35:30Z

@marmbrus I've fixed the linebreaks, but jenkins does not seem to pick up the changes and rerun the tests :-( What should I do?

marmbrus · 2015-04-25T21:38:29Z

test this please

rtreffer · 2015-04-27T09:00:25Z

@marmbrus what should we do now? New PR?

AmplabJenkins · 2015-04-27T18:18:30Z

Can one of the admins verify this patch?

marmbrus · 2015-04-27T18:22:05Z

Ok to test
On Apr 27, 2015 11:18 AM, "UCB AMPLab" notifications@github.com wrote:

Can one of the admins verify this patch?

—
Reply to this email directly or view it on GitHub
#5555 (comment).

rtreffer · 2015-04-30T08:14:26Z

Still no build 😢

marmbrus · 2015-04-30T16:37:24Z

Yeah sorry... jenkins was down i think.

marmbrus · 2015-04-30T16:37:29Z

ok to test

AmplabJenkins · 2015-04-30T16:42:10Z

Merged build triggered.

AmplabJenkins · 2015-04-30T16:42:20Z

Merged build started.

SparkQA · 2015-04-30T16:43:02Z

Test build #31431 has started for PR 5555 at commit 6a528d4.

SparkQA · 2015-04-30T16:44:58Z

Test build #31431 has finished for PR 5555 at commit 6a528d4.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-30T16:44:59Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-04-30T16:44:59Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31431/
Test FAILed.

AmplabJenkins · 2015-04-30T16:57:10Z

Merged build triggered.

AmplabJenkins · 2015-04-30T16:57:20Z

Merged build started.

SparkQA · 2015-04-30T16:58:53Z

Test build #31432 has started for PR 5555 at commit 5ab9be9.

SparkQA · 2015-04-30T18:38:55Z

Test build #31432 has finished for PR 5555 at commit 5ab9be9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-30T18:38:59Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-04-30T18:38:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31432/
Test PASSed.

AmplabJenkins · 2015-05-17T22:22:11Z

Merged build triggered.

AmplabJenkins · 2015-05-17T22:22:18Z

Merged build started.

SparkQA · 2015-05-17T22:23:28Z

Test build #32949 has started for PR 5555 at commit 5033611.

Replace the DriverQuirks with JdbcDialect(s) (and MySQLDialect/PostgresDialect) and allow developers to change the dialects on the fly (for new JDBCRRDs only). Some types (like an unsigned 64bit number) can be trivially mapped to java. The status quo is that the RRD will fail to load. This patch makes it possible to overwrite the type mapping to read e.g. 64Bit numbers as strings and handle them afterwards in software. JDBCSuite has an example that maps all types to String, which should always work (at the cost of extra code afterwards).

SparkQA · 2015-05-17T22:34:40Z

Test build #32949 has finished for PR 5555 at commit 5033611.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-17T22:34:43Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-17T22:34:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32949/
Test FAILed.

AmplabJenkins · 2015-05-17T22:37:13Z

Merged build triggered.

AmplabJenkins · 2015-05-17T22:37:23Z

Merged build started.

rtreffer · 2015-05-17T22:37:24Z

Rebased on master, squashed, added a case class for the JdbcType and got rid of null.

rtreffer · 2015-05-17T22:37:56Z

Jenkins failes with

[info] spark-sql: found 5 potential binary incompatibilities (filtered 287)
[error]  * object org.apache.spark.sql.jdbc.DriverQuirks does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.jdbc.DriverQuirks$")
[error]  * class org.apache.spark.sql.jdbc.DriverQuirks does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.jdbc.DriverQuirks")
[error]  * class org.apache.spark.sql.jdbc.PostgresQuirks does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.jdbc.PostgresQuirks")
[error]  * class org.apache.spark.sql.jdbc.NoQuirks does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.jdbc.NoQuirks")
[error]  * class org.apache.spark.sql.jdbc.MySQLQuirks does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.jdbc.MySQLQuirks")

but that was the intention of the rename.

rxin · 2015-05-17T22:40:25Z

@rtreffer you can change MimaExcludes.scala to include the exclude rules.

SparkQA · 2015-05-17T22:41:37Z

Test build #32951 has started for PR 5555 at commit fe7e2e8.

SparkQA · 2015-05-17T22:52:55Z

Test build #32951 has finished for PR 5555 at commit fe7e2e8.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

rtreffer · 2015-05-18T07:35:32Z

@rxin thank you, found the MimaExcludes, let's see if I've done that correctly.

AmplabJenkins · 2015-05-18T07:37:11Z

Merged build triggered.

AmplabJenkins · 2015-05-18T07:37:20Z

Merged build started.

SparkQA · 2015-05-18T07:39:20Z

Test build #32979 has started for PR 5555 at commit 3cbafd7.

SparkQA · 2015-05-18T09:59:26Z

Test build #32979 has finished for PR 5555 at commit 3cbafd7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-18T09:59:32Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-18T09:59:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32979/
Test PASSed.

marmbrus · 2015-05-18T18:57:22Z

Thanks! Merged to master and 1.4

Replace the DriverQuirks with JdbcDialect(s) (and MySQLDialect/PostgresDialect) and allow developers to change the dialects on the fly (for new JDBCRRDs only). Some types (like an unsigned 64bit number) can be trivially mapped to java. The status quo is that the RRD will fail to load. This patch makes it possible to overwrite the type mapping to read e.g. 64Bit numbers as strings and handle them afterwards in software. JDBCSuite has an example that maps all types to String, which should always work (at the cost of extra code afterwards). As a side effect it should now be possible to develop simple dialects out-of-tree and even with spark-shell. Author: Rene Treffer <treffer@measite.de> Closes #5555 from rtreffer/jdbc-dialects and squashes the following commits: 3cbafd7 [Rene Treffer] [SPARK-6888] ignore classes belonging to changed API in MIMA report fe7e2e8 [Rene Treffer] [SPARK-6888] Make the jdbc driver handling user-definable

Replace the DriverQuirks with JdbcDialect(s) (and MySQLDialect/PostgresDialect) and allow developers to change the dialects on the fly (for new JDBCRRDs only). Some types (like an unsigned 64bit number) can be trivially mapped to java. The status quo is that the RRD will fail to load. This patch makes it possible to overwrite the type mapping to read e.g. 64Bit numbers as strings and handle them afterwards in software. JDBCSuite has an example that maps all types to String, which should always work (at the cost of extra code afterwards). As a side effect it should now be possible to develop simple dialects out-of-tree and even with spark-shell. Author: Rene Treffer <treffer@measite.de> Closes apache#5555 from rtreffer/jdbc-dialects and squashes the following commits: 3cbafd7 [Rene Treffer] [SPARK-6888] ignore classes belonging to changed API in MIMA report fe7e2e8 [Rene Treffer] [SPARK-6888] Make the jdbc driver handling user-definable

rtreffer mentioned this pull request Apr 17, 2015

[SPARK-6888][SQL] Export driver quirks #5498

Closed

rtreffer force-pushed the jdbc-dialects branch from 3e98f74 to 3b3d6f6 Compare April 24, 2015 20:09

rtreffer force-pushed the jdbc-dialects branch from 3b3d6f6 to 6a528d4 Compare April 25, 2015 10:27

rtreffer force-pushed the jdbc-dialects branch from 5ab9be9 to 5033611 Compare May 17, 2015 22:20

rtreffer force-pushed the jdbc-dialects branch from 5033611 to fe7e2e8 Compare May 17, 2015 22:36

[SPARK-6888] ignore classes belonging to changed API in MIMA report

3cbafd7

asfgit closed this in e1ac2a9 May 18, 2015

Conversation

rtreffer commented Apr 17, 2015

Uh oh!

AmplabJenkins commented Apr 17, 2015

Uh oh!

marmbrus commented Apr 17, 2015

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

SparkQA commented Apr 17, 2015

Uh oh!

AmplabJenkins commented Apr 17, 2015

Uh oh!

marmbrus commented Apr 21, 2015

Uh oh!

rtreffer commented Apr 24, 2015

Uh oh!

rtreffer commented Apr 25, 2015

Uh oh!

marmbrus commented Apr 25, 2015

Uh oh!

rtreffer commented Apr 27, 2015

Uh oh!

AmplabJenkins commented Apr 27, 2015

Uh oh!

marmbrus commented Apr 27, 2015

Uh oh!

rtreffer commented Apr 30, 2015

Uh oh!

marmbrus commented Apr 30, 2015

Uh oh!

marmbrus commented Apr 30, 2015

Uh oh!

AmplabJenkins commented Apr 30, 2015

Uh oh!

AmplabJenkins commented Apr 30, 2015

Uh oh!

SparkQA commented Apr 30, 2015

Uh oh!

SparkQA commented Apr 30, 2015

Uh oh!

AmplabJenkins commented Apr 30, 2015

Uh oh!

AmplabJenkins commented Apr 30, 2015

Uh oh!

AmplabJenkins commented Apr 30, 2015

Uh oh!

AmplabJenkins commented Apr 30, 2015

Uh oh!

SparkQA commented Apr 30, 2015

Uh oh!

SparkQA commented Apr 30, 2015

Uh oh!

AmplabJenkins commented Apr 30, 2015

Uh oh!

AmplabJenkins commented Apr 30, 2015

Uh oh!

AmplabJenkins commented May 17, 2015

Uh oh!

AmplabJenkins commented May 17, 2015

Uh oh!

SparkQA commented May 17, 2015

Uh oh!

SparkQA commented May 17, 2015

Uh oh!

AmplabJenkins commented May 17, 2015

Uh oh!

AmplabJenkins commented May 17, 2015

Uh oh!

AmplabJenkins commented May 17, 2015

Uh oh!

AmplabJenkins commented May 17, 2015

Uh oh!

rtreffer commented May 17, 2015

Uh oh!

rtreffer commented May 17, 2015

Uh oh!

rxin commented May 17, 2015

Uh oh!

SparkQA commented May 17, 2015