Skip to content

[SPARK-15040][ML][PYSPARK] Add Imputer to PySpark - #17316

Closed
MLnick wants to merge 5 commits into
apache:masterfrom
MLnick:SPARK-15040-pyspark-imputer
Closed

[SPARK-15040][ML][PYSPARK] Add Imputer to PySpark#17316
MLnick wants to merge 5 commits into
apache:masterfrom
MLnick:SPARK-15040-pyspark-imputer

Conversation

@MLnick

@MLnick MLnick commented Mar 16, 2017

Copy link
Copy Markdown
Contributor

Add Python wrapper for Imputer feature transformer.

How was this patch tested?

New doc tests and tweak to PySpark ML tests.py

@MLnick

MLnick commented Mar 16, 2017

Copy link
Copy Markdown
Contributor Author

cc @hhbyyh

@SparkQA

SparkQA commented Mar 16, 2017

Copy link
Copy Markdown

Test build #74667 has finished for PR 17316 at commit 5efe889.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class Imputer(JavaEstimator, HasInputCols, JavaMLReadable, JavaMLWritable):
  • class ImputerModel(JavaModel, JavaMLReadable, JavaMLWritable):

@SparkQA

SparkQA commented Mar 16, 2017

Copy link
Copy Markdown

Test build #74669 has finished for PR 17316 at commit 5e53e05.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Mar 16, 2017

Copy link
Copy Markdown

Test build #74672 has finished for PR 17316 at commit 325c9cf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

hhbyyh
hhbyyh approved these changes Mar 16, 2017
View reviewed changes

@hhbyyh hhbyyh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Comment thread python/pyspark/ml/feature.py
"""
.. note:: Experimental

Imputation estimator for completing missing values, either using the mean or the median

@hhbyyh hhbyyh Mar 16, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Shall we change all the "column" to "columns" since we are supporting multiple columns now...

@MLnick MLnick Mar 17, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do for Python and Scala doc

Comment thread python/pyspark/ml/feature.py
def getOutputCols(self):
"""
Gets the value of :py:attr:`outputCols` or its default value.
"""

@hhbyyh hhbyyh Mar 16, 2017 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reminds me we should add

    require(get(inputCols).isDefined, "Input cols must be defined first.")
    require(get(outputCols).isDefined, "Output cols must be defined first.")

in transformschema

@MLnick MLnick Mar 17, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need that? The first call to $(inputCols) in validateAndTransformSchema will just throw an error with Failed to find a default value ...

@hhbyyh hhbyyh mentioned this pull request Mar 16, 2017
Closed
@SparkQA

SparkQA commented Mar 20, 2017

Copy link
Copy Markdown

Test build #74874 has finished for PR 17316 at commit 5c272b5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Mar 22, 2017

Copy link
Copy Markdown

Test build #75012 has finished for PR 17316 at commit 7fd17dd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MLnick

MLnick commented Mar 24, 2017

Copy link
Copy Markdown
Contributor Author

Merged to master.

@asfgit asfgit closed this in d9f4ce6 Mar 24, 2017
asfgit pushed a commit that referenced this pull request Apr 3, 2017
@YY-OnCall Nick Pentreath
[SPARK-19969][ML] Imputer doc and example
4d28e84 
## What changes were proposed in this pull request?

Add docs and examples for spark.ml.feature.Imputer. Currently scala and Java examples are included. Python example will be added after #17316

## How was this patch tested?

local doc generation and example execution

Author: Yuhao Yang <yuhao.yang@intel.com>

Closes #17324 from hhbyyh/imputerdoc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@hhbyyh hhbyyh hhbyyh approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@MLnick @SparkQA @hhbyyh