[SPARK-7378] [core] Handle deep links to unloaded apps. by vanzin · Pull Request #5922 · apache/spark

vanzin · 2015-05-05T21:32:26Z

The code was treating deep links as if they were attempt IDs, so
for example if you tried to load "/history/app1/jobs" directly,
that would fail because the code would treat "jobs" as an attempt id.

This change modifies the code to try both cases - first without an
attempt id, then with it, so that deep links are handled correctly.
This assumes that the links in the Spark UI do not clash with the
attempt id namespace, though, which is the case for YARN at least,
which is the only backend that currently publishes attempt IDs.

The code was treating deep links as if they were attempt IDs, so for example if you tried to load "/history/app1/jobs" directly, that would fail because the code would treat "jobs" as an attempt id. This change modifies the code to try both cases - first without an attempt id, then with it, so that deep links are handled correctly. This assumes that the links in the Spark UI do not clash with the attempt id namespace, though, which is the case for YARN at least, which is the only backend that currently publishes attempt IDs.

AmplabJenkins · 2015-05-05T21:37:09Z

Merged build triggered.

AmplabJenkins · 2015-05-05T21:37:14Z

Merged build started.

SparkQA · 2015-05-05T21:38:45Z

Test build #31905 has started for PR 5922 at commit b728f08.

SparkQA · 2015-05-05T21:40:31Z

Test build #31905 has finished for PR 5922 at commit b728f08.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-05T21:40:32Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-05T21:40:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31905/
Test FAILed.

AmplabJenkins · 2015-05-05T22:07:10Z

Merged build triggered.

AmplabJenkins · 2015-05-05T22:07:20Z

Merged build started.

SparkQA · 2015-05-05T22:09:01Z

Test build #31912 has started for PR 5922 at commit 23483e4.

SparkQA · 2015-05-05T23:21:03Z

Test build #31912 has finished for PR 5922 at commit 23483e4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-05T23:21:09Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-05T23:21:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31912/
Test FAILed.

vanzin · 2015-05-06T01:35:21Z

Streaming failure, sigh.

vanzin · 2015-05-06T01:35:28Z

Jenkins, retest this please.

AmplabJenkins · 2015-05-06T01:37:10Z

Merged build triggered.

AmplabJenkins · 2015-05-06T01:37:16Z

Merged build started.

SparkQA · 2015-05-06T01:39:03Z

Test build #31932 has started for PR 5922 at commit 23483e4.

SparkQA · 2015-05-06T01:53:07Z

Test build #31932 has finished for PR 5922 at commit 23483e4.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-06T01:53:10Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-06T01:53:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31932/
Test FAILed.

AmplabJenkins · 2015-05-06T17:12:11Z

Merged build triggered.

AmplabJenkins · 2015-05-06T17:12:19Z

Merged build started.

SparkQA · 2015-05-06T17:14:07Z

Test build #31996 has started for PR 5922 at commit ed3bcd4.

SparkQA · 2015-05-06T18:54:35Z

Test build #31996 has finished for PR 5922 at commit ed3bcd4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-06T18:54:40Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-06T18:54:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31996/
Test PASSed.

vanzin · 2015-05-07T19:26:11Z

@andrewor14 @srowen could you guys take a look at this? This is a regression from 1.3.

andrewor14 · 2015-05-07T19:32:54Z

should this be >= 3?

Good catch. Let me retest and update the patch.

AmplabJenkins · 2015-05-07T20:02:11Z

Merged build triggered.

AmplabJenkins · 2015-05-07T20:02:18Z

Merged build started.

SparkQA · 2015-05-07T20:04:06Z

Test build #32144 has started for PR 5922 at commit 96f648b.

SparkQA · 2015-05-07T21:45:51Z

Test build #32144 has finished for PR 5922 at commit 96f648b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-07T21:45:56Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-07T21:45:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32144/
Test PASSed.

srowen · 2015-05-08T09:01:01Z

LGTM

andrewor14 · 2015-05-08T21:12:23Z

LGTM master 1.4

The code was treating deep links as if they were attempt IDs, so for example if you tried to load "/history/app1/jobs" directly, that would fail because the code would treat "jobs" as an attempt id. This change modifies the code to try both cases - first without an attempt id, then with it, so that deep links are handled correctly. This assumes that the links in the Spark UI do not clash with the attempt id namespace, though, which is the case for YARN at least, which is the only backend that currently publishes attempt IDs. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #5922 from vanzin/SPARK-7378 and squashes the following commits: 96f648b [Marcelo Vanzin] Fix comparison. ed3bcd4 [Marcelo Vanzin] Merge branch 'master' into SPARK-7378 23483e4 [Marcelo Vanzin] Fat fingers. b728f08 [Marcelo Vanzin] [SPARK-7378] [core] Handle deep links to unloaded apps. (cherry picked from commit 5467c34) Signed-off-by: Andrew Or <andrew@databricks.com>

The code was treating deep links as if they were attempt IDs, so for example if you tried to load "/history/app1/jobs" directly, that would fail because the code would treat "jobs" as an attempt id. This change modifies the code to try both cases - first without an attempt id, then with it, so that deep links are handled correctly. This assumes that the links in the Spark UI do not clash with the attempt id namespace, though, which is the case for YARN at least, which is the only backend that currently publishes attempt IDs. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes apache#5922 from vanzin/SPARK-7378 and squashes the following commits: 96f648b [Marcelo Vanzin] Fix comparison. ed3bcd4 [Marcelo Vanzin] Merge branch 'master' into SPARK-7378 23483e4 [Marcelo Vanzin] Fat fingers. b728f08 [Marcelo Vanzin] [SPARK-7378] [core] Handle deep links to unloaded apps.

Fat fingers.

23483e4

Merge branch 'master' into SPARK-7378

ed3bcd4

andrewor14 reviewed May 7, 2015
View reviewed changes

Fix comparison.

96f648b

asfgit closed this in 5467c34 May 8, 2015

vanzin deleted the SPARK-7378 branch May 12, 2015 20:25

Conversation

vanzin commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

SparkQA commented May 5, 2015

Uh oh!

SparkQA commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

SparkQA commented May 5, 2015

Uh oh!

SparkQA commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

vanzin commented May 6, 2015

Uh oh!

vanzin commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

vanzin commented May 7, 2015

Uh oh!

andrewor14 May 7, 2015

Choose a reason for hiding this comment

Uh oh!

vanzin May 7, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

SparkQA commented May 7, 2015

Uh oh!

SparkQA commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

srowen commented May 8, 2015

Uh oh!

andrewor14 commented May 8, 2015

Uh oh!

Reviewers