Skip to content

[SPARK-6216] [PySpark] check python version of worker with driver - #6203

Closed
davies wants to merge 3 commits into
apache:masterfrom
davies:py_version
Closed

[SPARK-6216] [PySpark] check python version of worker with driver#6203
davies wants to merge 3 commits into
apache:masterfrom
davies:py_version

Conversation

@davies

@davies davies commented May 15, 2015

Copy link
Copy Markdown
Contributor

This PR revert #5404, change to pass the version of python in driver into JVM, check it in worker before deserializing closure, then it can works with different major version of Python.

@davies

davies commented May 15, 2015

Copy link
Copy Markdown
Contributor Author

cc @JoshRosen

@AmplabJenkins

AmplabJenkins commented May 15, 2015

Copy link
Copy Markdown

Merged build triggered.

@AmplabJenkins

AmplabJenkins commented May 15, 2015

Copy link
Copy Markdown

Merged build started.

@SparkQA

SparkQA commented May 15, 2015

Copy link
Copy Markdown

Test build #32859 has started for PR 6203 at commit 47c6278.

vanzin
vanzin reviewed May 15, 2015
View reviewed changes
Comment thread python/pyspark/context.py Outdated

@vanzin vanzin May 15, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not a huge deal, but it might be better to pass major and minor as separate parameters. You know, in case python 2.10 ever comes out. :-)

Or, at least, use a larger multiplier.

@vanzin

vanzin commented May 15, 2015

Copy link
Copy Markdown
Contributor

LGTM aside from minor suggestion.

@SparkQA

SparkQA commented May 16, 2015

Copy link
Copy Markdown

Test build #32859 has finished for PR 6203 at commit 47c6278.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented May 16, 2015

Copy link
Copy Markdown

Merged build finished. Test FAILed.

@AmplabJenkins

AmplabJenkins commented May 16, 2015

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32859/
Test FAILed.

@AmplabJenkins

AmplabJenkins commented May 16, 2015

Copy link
Copy Markdown

Merged build triggered.

@AmplabJenkins

AmplabJenkins commented May 16, 2015

Copy link
Copy Markdown

Merged build started.

@SparkQA

SparkQA commented May 16, 2015

Copy link
Copy Markdown

Test build #32870 has started for PR 6203 at commit 6ce5096.

@SparkQA

SparkQA commented May 16, 2015

Copy link
Copy Markdown

Test build #32870 has finished for PR 6203 at commit 6ce5096.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented May 16, 2015

Copy link
Copy Markdown

Merged build finished. Test FAILed.

@AmplabJenkins

AmplabJenkins commented May 16, 2015

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32870/
Test FAILed.

@AmplabJenkins

AmplabJenkins commented May 16, 2015

Copy link
Copy Markdown

Merged build triggered.

@AmplabJenkins

AmplabJenkins commented May 16, 2015

Copy link
Copy Markdown

Merged build started.

@SparkQA

SparkQA commented May 16, 2015

Copy link
Copy Markdown

Test build #32883 has started for PR 6203 at commit b8fb76e.

@SparkQA

SparkQA commented May 16, 2015

Copy link
Copy Markdown

Test build #32883 has finished for PR 6203 at commit b8fb76e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented May 16, 2015

Copy link
Copy Markdown

Merged build finished. Test PASSed.

@AmplabJenkins

AmplabJenkins commented May 16, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32883/
Test PASSed.

@JoshRosen

JoshRosen commented May 17, 2015

Copy link
Copy Markdown
Contributor

LGTM as well; this seems like a straightforward fix and will be useful once more people start playing around with Python 3.

@JoshRosen

JoshRosen commented May 18, 2015

Copy link
Copy Markdown
Contributor

I'm going to merge this into master and branch-1.4 (1.4.0). Thanks!

asfgit pushed a commit that referenced this pull request May 18, 2015
Davies Liu @JoshRosen
[SPARK-6216] [PYSPARK] check python version of worker with driver
a833209 
This PR revert #5404, change to pass the version of python in driver into JVM, check it in worker before deserializing closure, then it can works with different major version of Python.

Author: Davies Liu <davies@databricks.com>

Closes #6203 from davies/py_version and squashes the following commits:

b8fb76e [Davies Liu] fix test
6ce5096 [Davies Liu] use string for version
47c6278 [Davies Liu] check python version of worker with driver

(cherry picked from commit 32fbd29)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
@asfgit asfgit closed this in 32fbd29 May 18, 2015
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
Davies Liu @jeanlyn
[SPARK-6216] [PYSPARK] check python version of worker with driver
6ee9228 
This PR revert apache#5404, change to pass the version of python in driver into JVM, check it in worker before deserializing closure, then it can works with different major version of Python.

Author: Davies Liu <davies@databricks.com>

Closes apache#6203 from davies/py_version and squashes the following commits:

b8fb76e [Davies Liu] fix test
6ce5096 [Davies Liu] use string for version
47c6278 [Davies Liu] check python version of worker with driver
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
Davies Liu @jeanlyn
[SPARK-6216] [PYSPARK] check python version of worker with driver
f58e5b5 
This PR revert apache#5404, change to pass the version of python in driver into JVM, check it in worker before deserializing closure, then it can works with different major version of Python.

Author: Davies Liu <davies@databricks.com>

Closes apache#6203 from davies/py_version and squashes the following commits:

b8fb76e [Davies Liu] fix test
6ce5096 [Davies Liu] use string for version
47c6278 [Davies Liu] check python version of worker with driver
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@davies @AmplabJenkins @SparkQA @vanzin @JoshRosen