[SPARK-4423] Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior by ilganeli · Pull Request #4696 · apache/spark

ilganeli · 2015-02-19T18:14:15Z

Hi all - I've added a writeup on how closures work within Spark to help clarify the general case for this problem and similar problems. I hope this addresses the issue and would love any feedback.

…tand confusing behavior of foreach and map functions when attempting to modify variables outside of the scope of an RDD action or transformation

SparkQA · 2015-02-19T18:17:37Z

Test build #27728 has started for PR 4696 at commit d374d3a.

This patch merges cleanly.

SparkQA · 2015-02-19T19:33:50Z

Test build #27728 has finished for PR 4696 at commit d374d3a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-19T19:33:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27728/
Test PASSed.

rxin · 2015-02-20T08:14:01Z

probably better to word it as "It is important to understand the scope and life cycle of variables and methods ...", instead of saying it is unintuitive.

Also overall I think you'd want to point out closures are always executed on executors and should not be used to mutate state, and state that the only exception is when running in local testing mode. If some global aggregation is needed, use an aggregator.

ilganeli · 2015-02-20T17:12:48Z

Hi @rxin, thanks for the feedback. I've updated the doc, please let me know what you think.

SparkQA · 2015-02-20T17:17:36Z

Test build #27779 has started for PR 4696 at commit 5dbbda5.

This patch merges cleanly.

SparkQA · 2015-02-20T18:35:33Z

Test build #27779 has finished for PR 4696 at commit 5dbbda5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-20T18:35:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27779/
Test PASSed.

JoshRosen · 2015-02-20T20:46:39Z

Typo: naiive (should only have one i)

I think the comma here should come after "below"

ilganeli · 2015-02-20T22:32:48Z

Hi Josh - I pulled in your suggestions. Please let me know if anything else needs work. Thanks!

SparkQA · 2015-02-20T22:37:35Z

Test build #27794 has started for PR 4696 at commit 448bd79.

This patch merges cleanly.

SparkQA · 2015-02-20T23:55:53Z

Test build #27794 has finished for PR 4696 at commit 448bd79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-20T23:55:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27794/
Test PASSed.

ilganeli · 2015-02-24T23:42:00Z

Hi all - does this need anything else prior to being merged?

srowen · 2015-03-02T13:29:50Z

I think it's worth sharpening what "local" means -- master = local[n] right? It's not specific to the shell; you can run the shell against YARN.

Really the difference is between happening to execute entirely within one JVM, and not. It is possible to run a standalone cluster "locally" that would not exhibit this behavior.

srowen · 2015-03-02T13:45:27Z

Same, I'd describe this as undefined instead. It will not necessarily update local variables in the driver and should be avoided.

I think this should still be updated, and should probably make a few edits like spark -> Spark, or accumulator -> Accumulator

SparkQA · 2015-03-09T20:02:51Z

Test build #28400 has started for PR 4696 at commit 4772f99.

This patch merges cleanly.

SparkQA · 2015-03-09T21:18:00Z

Test build #28400 has finished for PR 4696 at commit 4772f99.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-09T21:18:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28400/
Test PASSed.

srowen · 2015-03-09T21:34:40Z

This one won't compile and I think the Python equivalent won't either, but don't fix it up yet; if that's all there is I can easily do it on merge.

srowen · 2015-03-09T21:47:53Z

Yep, I think it does need one more pass of edits but is 99% there.

… semicolons for java

SparkQA · 2015-03-10T20:12:43Z

Test build #28437 has started for PR 4696 at commit 2fd2a07.

This patch merges cleanly.

SparkQA · 2015-03-10T21:33:17Z

Test build #28437 has finished for PR 4696 at commit 2fd2a07.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-10T21:33:21Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28437/
Test PASSed.

srowen · 2015-03-10T21:37:09Z

Looking good but there is still a set of small changes I think need to be made in the last diff in this PR to be consistent with the rest of the text. See my notes there. Otherwise LGTM and will leave it open for a day for final comments.

ilganeli · 2015-03-10T21:47:57Z

Hi Sean - I've added those fixes. I missed that last set of comments you mentioned. Thanks.

SparkQA · 2015-03-10T21:52:47Z

Test build #28443 has started for PR 4696 at commit c5dc498.

This patch merges cleanly.

AmplabJenkins · 2015-03-10T22:02:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28442/
Test FAILed.

SparkQA · 2015-03-10T23:13:29Z

Test build #28443 has finished for PR 4696 at commit c5dc498.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-10T23:13:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28443/
Test PASSed.

Ilya Ganelin added 3 commits February 12, 2015 08:20

Updated documentation to add a section on closures. This helps unders…

c768ab2

…tand confusing behavior of foreach and map functions when attempting to modify variables outside of the scope of an RDD action or transformation

Minor edits

2600668

Merge remote-tracking branch 'upstream/master' into SPARK-4423

d374d3a

rxin reviewed Feb 20, 2015
View reviewed changes

Improved some wording

5dbbda5

JoshRosen reviewed Feb 20, 2015
View reviewed changes

Updated some verbage and added section links

448bd79

srowen reviewed Mar 2, 2015
View reviewed changes

Incorporated latest feedback

4772f99

srowen reviewed Mar 9, 2015
View reviewed changes

Incoporated a few more minor fixes. Fixed a bug in python code. Added…

2fd2a07

… semicolons for java

Ilya Ganelin added 2 commits March 10, 2015 14:46

Updated to fix capitalization and clarify wording

48c1983

Updated to fix capitalization

07b78e8

Fixed typo

c5dc498

asfgit closed this in 548643a Mar 11, 2015

Conversation

ilganeli commented Feb 19, 2015

Uh oh!

SparkQA commented Feb 19, 2015

Uh oh!

SparkQA commented Feb 19, 2015

Uh oh!

AmplabJenkins commented Feb 19, 2015

Uh oh!

rxin Feb 20, 2015

Choose a reason for hiding this comment

Uh oh!

ilganeli commented Feb 20, 2015

Uh oh!

SparkQA commented Feb 20, 2015

Uh oh!

SparkQA commented Feb 20, 2015

Uh oh!

AmplabJenkins commented Feb 20, 2015

Uh oh!

JoshRosen Feb 20, 2015

Choose a reason for hiding this comment

Uh oh!

JoshRosen Feb 20, 2015

Choose a reason for hiding this comment

Uh oh!

ilganeli commented Feb 20, 2015

Uh oh!

SparkQA commented Feb 20, 2015

Uh oh!

SparkQA commented Feb 20, 2015

Uh oh!

AmplabJenkins commented Feb 20, 2015

Uh oh!

ilganeli commented Feb 24, 2015

Uh oh!

srowen Mar 2, 2015

Choose a reason for hiding this comment

Uh oh!

srowen Mar 2, 2015

Choose a reason for hiding this comment

Uh oh!

srowen Mar 9, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 9, 2015

Uh oh!

SparkQA commented Mar 9, 2015

Uh oh!

AmplabJenkins commented Mar 9, 2015

Uh oh!

srowen Mar 9, 2015

Choose a reason for hiding this comment

Uh oh!

srowen commented Mar 9, 2015

Uh oh!

SparkQA commented Mar 10, 2015

Uh oh!

SparkQA commented Mar 10, 2015

Uh oh!

AmplabJenkins commented Mar 10, 2015

Uh oh!

srowen commented Mar 10, 2015

Uh oh!

ilganeli commented Mar 10, 2015

Uh oh!

SparkQA commented Mar 10, 2015

Uh oh!

AmplabJenkins commented Mar 10, 2015

Uh oh!

SparkQA commented Mar 10, 2015

Uh oh!

AmplabJenkins commented Mar 10, 2015

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone