[SPARK-1516]Throw exception in yarn client instead of run system.exit #1099

codeboyyong · 2014-06-16T23:05:06Z

[SPARK-1516]Throw exception in yarn client instead of run system.exit directly.

All the changes is in  the package of "org.apache.spark.deploy.yarn":
    1) Throw IllegalArgumentException in ClinetArguments  instead of exit directly.
    2) In Client's main method, if exception is caught, it will exit with code 1, otherwise exit with code 0.
    3) In YarnClientSchedulerBackend's start method, if IllegalArgumentException is caught, it will exit with code 1, otherwise throw that exception.
    4) Fix some message typo in the Client.scala
After the fix, if user integrate the spark yarn client into their applications,
when the argument is wrong or the running is finished, the application won't be terminated.

+CC @dbtsai @mengxr

AmplabJenkins · 2014-06-16T23:09:46Z

Merged build triggered.

AmplabJenkins · 2014-06-16T23:09:51Z

Merged build started.

AmplabJenkins · 2014-06-16T23:40:42Z

Merged build finished.

AmplabJenkins · 2014-06-16T23:40:42Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15827/

mengxr · 2014-06-17T04:55:58Z

yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala

+        Console.err.println(e.getMessage)
+        System.exit(1)
+      }
+      case x:Throwable => throw x


Remove this line since this is the default behavior.

mengxr · 2014-06-17T05:01:23Z

@codeboyyong Thanks for submitting the patch! It looks good to me except a few style issues. Did you test both stable and alpha yarn clusters?

…t directly. All the changes is in the package of "org.apache.spark.deploy.yarn": 1) Throw IllegalArgumentException in ClinetArguments instead of exit directly. 2) In Client's main method, if exception is caught, it will exit with code 1, otherwise exit with code 0. 3) In YarnClientSchedulerBackend's start method, if IllegalArgumentException is caught, it will exit with code 1, otherwise throw that exception. 4) Fix some message typo in the Client.scala After the fix, if user integrate the spark yarn client into their applications, when the argument is wrong or the running is finished, the application won't be terminated.

AmplabJenkins · 2014-06-17T06:34:47Z

Merged build triggered.

AmplabJenkins · 2014-06-17T06:34:54Z

Merged build started.

AmplabJenkins · 2014-06-17T07:02:35Z

Merged build finished.

AmplabJenkins · 2014-06-17T07:02:35Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15845/

codeboyyong · 2014-06-17T07:39:05Z

@mengxr . I fix the format . And I did the test on hadoop-2.0.5-alpha and hadoop 2.2.0. The command line now works as usual. And my local yarn cluster is not working .I might test the run time exit later. Will let you know the result

…" so that client console can get more useful information when somthing is wrong.

AmplabJenkins · 2014-06-17T22:39:49Z

Merged build triggered.

AmplabJenkins · 2014-06-17T22:39:55Z

Merged build started.

AmplabJenkins · 2014-06-17T23:07:32Z

Merged build finished.

AmplabJenkins · 2014-06-17T23:07:32Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15857/

mengxr · 2014-06-23T07:44:22Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-23T07:45:10Z

Merged build triggered.

AmplabJenkins · 2014-06-23T07:45:16Z

Merged build started.

AmplabJenkins · 2014-06-23T08:13:13Z

Merged build finished.

AmplabJenkins · 2014-06-23T08:13:13Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16019/

mengxr · 2014-06-24T01:59:10Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-24T02:00:14Z

Merged build triggered.

AmplabJenkins · 2014-06-24T02:00:23Z

Merged build started.

AmplabJenkins · 2014-06-24T02:29:02Z

Merged build finished.

mengxr · 2014-06-26T06:01:19Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-26T06:05:21Z

Merged build triggered.

AmplabJenkins · 2014-06-26T06:05:27Z

Merged build started.

AmplabJenkins · 2014-06-26T06:30:51Z

Merged build finished.

AmplabJenkins · 2014-06-26T06:30:51Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16138/

tgravescs · 2014-06-26T16:13:27Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-26T16:15:28Z

Merged build triggered.

AmplabJenkins · 2014-06-26T16:15:36Z

Merged build started.

AmplabJenkins · 2014-06-26T16:48:51Z

Merged build finished.

AmplabJenkins · 2014-06-26T16:48:51Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16161/

dbtsai · 2014-06-26T16:50:18Z

Seems that the jenkins is missing the python runtime.

codeboyyong · 2014-06-26T18:21:05Z

Would this be caused by this commit?
Should we revert it and try again ?

John.

On Jun 26, 2014, at 9:50 AM, DB Tsai notifications@github.com wrote:

Seems that the jenkins is missing the python runtime.

—
Reply to this email directly or view it on GitHub.

mengxr · 2014-06-26T23:04:15Z

I think PYSPARK_PYTHON is set to /usr/local/bin/python2.7 in Jenkins but it doesn't exist. @pwendell ?

mengxr · 2014-07-03T18:38:24Z

Jenkins, retest this please.

AmplabJenkins · 2014-07-03T18:40:50Z

Merged build triggered.

AmplabJenkins · 2014-07-03T18:40:57Z

Merged build started.

AmplabJenkins · 2014-07-03T20:07:05Z

Merged build finished.

AmplabJenkins · 2014-07-03T20:07:05Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16310/

mengxr · 2014-07-03T20:42:24Z

Jenkins, retest this please.

AmplabJenkins · 2014-07-03T20:45:49Z

Merged build triggered.

AmplabJenkins · 2014-07-03T20:45:57Z

Merged build started.

AmplabJenkins · 2014-07-03T22:14:44Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-07-03T22:14:44Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16312/

mengxr · 2014-07-03T22:18:24Z

Merged. Thanks!

[SPARK-1516]Throw exception in yarn client instead of run system.exit directly. All the changes is in the package of "org.apache.spark.deploy.yarn": 1) Throw IllegalArgumentException in ClinetArguments instead of exit directly. 2) In Client's main method, if exception is caught, it will exit with code 1, otherwise exit with code 0. 3) In YarnClientSchedulerBackend's start method, if IllegalArgumentException is caught, it will exit with code 1, otherwise throw that exception. 4) Fix some message typo in the Client.scala After the fix, if user integrate the spark yarn client into their applications, when the argument is wrong or the running is finished, the application won't be terminated. +CC dbtsai mengxr Author: John Zhao <codeboyyong@gmail.com> Closes #1099 from codeboyyong/branch-0.9 and squashes the following commits: 00144b5 [John Zhao] use e.printStackTrace() to replace "Console.err.println(e.getMessage)" so that client console can get more useful information when somthing is wrong. addcecb [John Zhao] [SPARK-1516]Throw exception in yarn client instead of run system.exit directly.

mengxr · 2014-07-07T22:10:33Z

@codeboyyong The PR was not closed automatically. Could you close this PR?

codeboyyong · 2014-07-08T00:13:59Z

Closed as xiangrui required.

…1099) * [CARMEL-6174][FOLLOWUP] Change prefer shuffled hash join condition * Select SHJ the max partition size than ADAPTIVE_SHUFFLE_HASH_JOIN_ADVISORY_STREAM_PARTITION_SIZE

mengxr reviewed Jun 17, 2014
View reviewed changes

use e.printStackTrace() to replace "Console.err.println(e.getMessage)…

00144b5

…" so that client console can get more useful information when somthing is wrong.

codeboyyong closed this Jul 8, 2014

[SPARK-1516]Throw exception in yarn client instead of run system.exit #1099

[SPARK-1516]Throw exception in yarn client instead of run system.exit #1099

Conversation

codeboyyong commented Jun 16, 2014

AmplabJenkins commented Jun 16, 2014

AmplabJenkins commented Jun 16, 2014

AmplabJenkins commented Jun 16, 2014

AmplabJenkins commented Jun 16, 2014

mengxr Jun 17, 2014

Choose a reason for hiding this comment

mengxr commented Jun 17, 2014

AmplabJenkins commented Jun 17, 2014

AmplabJenkins commented Jun 17, 2014

AmplabJenkins commented Jun 17, 2014

AmplabJenkins commented Jun 17, 2014

codeboyyong commented Jun 17, 2014

AmplabJenkins commented Jun 17, 2014

AmplabJenkins commented Jun 17, 2014

AmplabJenkins commented Jun 17, 2014

AmplabJenkins commented Jun 17, 2014

mengxr commented Jun 23, 2014

AmplabJenkins commented Jun 23, 2014

AmplabJenkins commented Jun 23, 2014

AmplabJenkins commented Jun 23, 2014

AmplabJenkins commented Jun 23, 2014

mengxr commented Jun 24, 2014

AmplabJenkins commented Jun 24, 2014

AmplabJenkins commented Jun 24, 2014

AmplabJenkins commented Jun 24, 2014

mengxr commented Jun 26, 2014

AmplabJenkins commented Jun 26, 2014

AmplabJenkins commented Jun 26, 2014

AmplabJenkins commented Jun 26, 2014

AmplabJenkins commented Jun 26, 2014

tgravescs commented Jun 26, 2014

AmplabJenkins commented Jun 26, 2014

AmplabJenkins commented Jun 26, 2014

AmplabJenkins commented Jun 26, 2014

AmplabJenkins commented Jun 26, 2014

dbtsai commented Jun 26, 2014

codeboyyong commented Jun 26, 2014

mengxr commented Jun 26, 2014

mengxr commented Jul 3, 2014

AmplabJenkins commented Jul 3, 2014

AmplabJenkins commented Jul 3, 2014

AmplabJenkins commented Jul 3, 2014

AmplabJenkins commented Jul 3, 2014

mengxr commented Jul 3, 2014

AmplabJenkins commented Jul 3, 2014

AmplabJenkins commented Jul 3, 2014

AmplabJenkins commented Jul 3, 2014

AmplabJenkins commented Jul 3, 2014

mengxr commented Jul 3, 2014

mengxr commented Jul 7, 2014

codeboyyong commented Jul 8, 2014