Skip to content

[SPARK-6055] [PySpark] fix incorrect DataType.__eq__ (for 1.2) - #4809

Closed
davies wants to merge 4 commits into
apache:branch-1.2from
davies:leak2
Closed

[SPARK-6055] [PySpark] fix incorrect DataType.__eq__ (for 1.2)#4809
davies wants to merge 4 commits into
apache:branch-1.2from
davies:leak2

Conversation

@davies

@davies davies commented Feb 27, 2015

Copy link
Copy Markdown
Contributor

The eq of DataType is not correct, class cache is not use correctly (created class can not be find by dataType), then it will create lots of classes (saved in _cached_cls), never released.

Also, all same DataType have same hash code, there will be many object in a dict with the same hash code, end with hash attach, it's very slow to access this dict (depends on the implementation of CPython).

This PR also improve the performance of inferSchema (avoid the unnecessary converter of object).

@davies davies changed the title [SPARK-6055] [PySpark] fix incorrect DataType.__eq__ [SPARK-6055] [PySpark] fix incorrect DataType.__eq__ (for 1.1/1.2) Feb 27, 2015
@SparkQA

SparkQA commented Feb 27, 2015

Copy link
Copy Markdown

Test build #28054 has started for PR 4809 at commit 6c2909a.

  • This patch merges cleanly.

@davies davies changed the title [SPARK-6055] [PySpark] fix incorrect DataType.__eq__ (for 1.1/1.2) [SPARK-6055] [PySpark] fix incorrect DataType.__eq__ (for 1.2) Feb 27, 2015
@SparkQA

SparkQA commented Feb 27, 2015

Copy link
Copy Markdown

Test build #28054 has finished for PR 4809 at commit 6c2909a.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Feb 27, 2015

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28054/
Test FAILed.

@SparkQA

SparkQA commented Feb 27, 2015

Copy link
Copy Markdown

Test build #28074 has started for PR 4809 at commit b576107.

  • This patch merges cleanly.

@SparkQA

SparkQA commented Feb 27, 2015

Copy link
Copy Markdown

Test build #28074 has finished for PR 4809 at commit b576107.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Feb 27, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28074/
Test PASSed.

@SparkQA

SparkQA commented Feb 27, 2015

Copy link
Copy Markdown

Test build #28086 has started for PR 4809 at commit 9b4dadc.

  • This patch merges cleanly.

@SparkQA

SparkQA commented Feb 27, 2015

Copy link
Copy Markdown

Test build #28093 has started for PR 4809 at commit 65c222f.

  • This patch merges cleanly.

JoshRosen
JoshRosen reviewed Feb 27, 2015
View reviewed changes
Comment thread python/pyspark/sql.py

@JoshRosen JoshRosen Feb 27, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that the PR opened against master added typechecking asserts here. Should we also add them in this branch, or is there a reason why we should omit them here?

@davies davies Feb 27, 2015

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we can not cherry-pick the patch from master, I need to re-do all the things on 1.2/1.1, so I'd like to keep the changes minimized.

@JoshRosen JoshRosen Feb 27, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough; just wanted to check.

@SparkQA

SparkQA commented Feb 27, 2015

Copy link
Copy Markdown

Test build #28086 has finished for PR 4809 at commit 9b4dadc.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Feb 27, 2015

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28086/
Test FAILed.

@JoshRosen

JoshRosen commented Feb 27, 2015

Copy link
Copy Markdown
Contributor

LGTM, since the changes here are a subset of the changes in the PR opened against master.

@SparkQA

SparkQA commented Feb 27, 2015

Copy link
Copy Markdown

Test build #28093 has finished for PR 4809 at commit 65c222f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Feb 27, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28093/
Test PASSed.

@JoshRosen

JoshRosen commented Feb 28, 2015

Copy link
Copy Markdown
Contributor

I've merged this into branch-1.2 (1.2.2). Thanks!

asfgit pushed a commit that referenced this pull request Feb 28, 2015
Davies Liu @JoshRosen
[SPARK-6055] [PySpark] fix incorrect DataType.__eq__ (for 1.2)
576fc54 
The eq of DataType is not correct, class cache is not use correctly (created class can not be find by dataType), then it will create lots of classes (saved in _cached_cls), never released.

Also, all same DataType have same hash code, there will be many object in a dict with the same hash code, end with hash attach, it's very slow to access this dict (depends on the implementation of CPython).

This PR also improve the performance of inferSchema (avoid the unnecessary converter of object).

Author: Davies Liu <davies@databricks.com>

Closes #4809 from davies/leak2 and squashes the following commits:

65c222f [Davies Liu] Update sql.py
9b4dadc [Davies Liu] fix __eq__ of singleton
b576107 [Davies Liu] fix tests
6c2909a [Davies Liu] fix incorrect DataType.__eq__
@pwendell

pwendell commented Mar 1, 2015

Copy link
Copy Markdown
Contributor

@davies can you close this? (auto close doesn't work for the backport commits).

@davies davies closed this Mar 1, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@davies @SparkQA @AmplabJenkins @JoshRosen @pwendell