Skip to content

[SPARK-8099] set executor cores into system in yarn-cluster mode - #6643

Closed
XuTingjun wants to merge 4 commits into
apache:masterfrom
XuTingjun:SPARK-8099
Closed

XuTingjun wants to merge 4 commits into
apache:masterfrom
XuTingjun:SPARK-8099

Conversation

@XuTingjun

Copy link
Copy Markdown
Contributor

No description provided.

@XuTingjun XuTingjun changed the title set executor cores into system [SPARK-8099] set executor cores into system in yarn-cluster mode Jun 4, 2015
@srowen

srowen commented Jun 4, 2015

Copy link
Copy Markdown
Member

This doesn't look like the right way to do it -- just manually setting a system property here? CC @tgravescs @sryza

@sryza

sryza commented Jun 4, 2015

Copy link
Copy Markdown
Contributor

I think the right place to fix this is probably https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L449.

We probably want it to look like the line below that sets spark.executor.memory for all cluster managers.

@XuTingjun

Copy link
Copy Markdown
Contributor Author

Thanks all, I have updated the code, please have a look again.

@srowen

srowen commented Jun 5, 2015

Copy link
Copy Markdown
Member

ok to test

@SparkQA

SparkQA commented Jun 5, 2015

Copy link
Copy Markdown

Test build #34259 has finished for PR 6643 at commit 80b18cd.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@XuTingjun

Copy link
Copy Markdown
Contributor Author

please retest

@srowen

srowen commented Jun 5, 2015

Copy link
Copy Markdown
Member

Jenkins, retest this please

@SparkQA

SparkQA commented Jun 5, 2015

Copy link
Copy Markdown

Test build #34275 has finished for PR 6643 at commit 80b18cd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen

srowen commented Jun 5, 2015

Copy link
Copy Markdown
Member

@sryza if that's what you meant you can merge it

@andrewor14

Copy link
Copy Markdown
Contributor

lgtm, great catch

@sryza

sryza commented Jun 5, 2015

Copy link
Copy Markdown
Contributor

LGTM, merging

@asfgit asfgit closed this in 0992a0a Jun 5, 2015
@XuTingjun
XuTingjun deleted the SPARK-8099 branch June 6, 2015 02:06
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
Author: Xutingjun <xutingjun@huawei.com>
Author: xutingjun <xutingjun@huawei.com>

Closes apache#6643 from XuTingjun/SPARK-8099 and squashes the following commits:

80b18cd [Xutingjun] change to STANDALONE | YARN
ce33148 [Xutingjun] set executor cores into system
e51cc9e [Xutingjun] set executor cores into system
0600861 [xutingjun] set executor cores into system
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
Author: Xutingjun <xutingjun@huawei.com>
Author: xutingjun <xutingjun@huawei.com>

Closes apache#6643 from XuTingjun/SPARK-8099 and squashes the following commits:

80b18cd [Xutingjun] change to STANDALONE | YARN
ce33148 [Xutingjun] set executor cores into system
e51cc9e [Xutingjun] set executor cores into system
0600861 [xutingjun] set executor cores into system
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants