[SPARK-6955] Perform port retries at NettyBlockTransferService level #5575

aarondav · 2015-04-18T17:56:57Z

Currently we're doing port retries in the TransportServer level, but this is not specified by the TransportContext API and it has other further-reaching impacts like causing undesirable behavior for the Yarn and Standalone shuffle services.

SparkQA · 2015-04-18T17:58:30Z

Test build #30526 has started for PR 5575 at commit c671f6f.

SparkQA · 2015-04-18T17:58:36Z

Test build #30526 has finished for PR 5575 at commit c671f6f.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-18T17:58:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30526/
Test FAILed.

aarondav · 2015-04-18T18:02:23Z

cc @SaintBacchus @andrewor13 @vanzin This patch aims to resolve SPARK-6955 while keeping the fix for SPARK-5444.

aarondav · 2015-04-18T18:03:06Z

err, cc @andrewor14, not @andrewor13.

SparkQA · 2015-04-18T18:03:32Z

Test build #30527 has started for PR 5575 at commit a3dc42d.

SparkQA · 2015-04-18T20:33:33Z

Test build #30527 timed out for PR 5575 at commit a3dc42d after a configured wait of 150m.

AmplabJenkins · 2015-04-18T20:33:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30527/
Test FAILed.

aarondav · 2015-04-19T06:07:10Z

Jenkins, retest this please.

SparkQA · 2015-04-19T06:13:41Z

Test build #30548 has started for PR 5575 at commit a3dc42d.

SparkQA · 2015-04-19T08:43:43Z

Test build #30548 timed out for PR 5575 at commit a3dc42d after a configured wait of 150m.

AmplabJenkins · 2015-04-19T08:43:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30548/
Test FAILed.

aarondav · 2015-04-19T19:26:06Z

Jenkins, retest this please.

SparkQA · 2015-04-19T19:28:42Z

Test build #30568 has started for PR 5575 at commit a3dc42d.

SparkQA · 2015-04-19T21:58:44Z

Test build #30568 timed out for PR 5575 at commit a3dc42d after a configured wait of 150m.

AmplabJenkins · 2015-04-19T21:58:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30568/
Test FAILed.

SparkQA · 2015-04-19T22:58:36Z

Test build #30569 has started for PR 5575 at commit df60b80.

SparkQA · 2015-04-20T00:43:45Z

Test build #30569 has finished for PR 5575 at commit df60b80.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

AmplabJenkins · 2015-04-20T00:43:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30569/
Test PASSed.

SaintBacchus · 2015-04-20T02:10:51Z

I seems to be a better way. Thanks for taking time to solve it.

vanzin · 2015-04-20T17:49:08Z

core/src/test/scala/org/apache/spark/network/netty/NettyBlockTransferServiceSuite.scala

+    val port = 17634
+    service0 = createService(port)
+    service0.port should be >= port
+    service0.port should be <= (port + 10) // avoid testing equality in case of simultaneous tests


This seems flaky. Isn't the previous check enough? Otherwise, instead of 10 I'd use 1024 which is the actual limit in the code.

I can increase the number no problem, though I guess we should use Utils.portMaxRetries instead. However, note that even 10 should be very unlikely to be hit, that would mean there's 5-10 concurrently running tests at this exact part of the build, or else we're very, very unlucky.

vanzin · 2015-04-20T17:51:16Z

LGTM aside from the potentially flaky test.

andrewor14 · 2015-04-21T00:03:22Z

core/src/test/scala/org/apache/spark/ExternalShuffleServiceSuite.scala

@@ -43,6 +43,9 @@ class ExternalShuffleServiceSuite extends ShuffleSuite with BeforeAndAfterAll {
    conf.set("spark.shuffle.manager", "sort")
    conf.set("spark.shuffle.service.enabled", "true")
    conf.set("spark.shuffle.service.port", server.getPort.toString)
+
+    // local-cluster mode starts a Worker which would start its own shuffle service without this:
+    conf.set("spark.worker.shouldHostShuffleServiceIfEnabled", "false")


Do we actually ever want to start an external shuffle service in a local cluster? If not I think it makes more sense to just set spark.shuffle.service.enabled to false in LocalSparkCluster (we already do this for the REST submission server for Master)

SparkQA · 2015-04-27T18:18:59Z

Test build #31002 has started for PR 5575 at commit df60b80.

AmplabJenkins · 2015-05-01T16:02:11Z

Merged build triggered.

SparkQA · 2015-05-01T18:34:02Z

Test build #31568 timed out for PR 5575 at commit d38075b after a configured wait of 150m.

AmplabJenkins · 2015-05-01T18:34:06Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-01T18:34:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31568/
Test FAILed.

AmplabJenkins · 2015-05-01T18:57:11Z

Merged build triggered.

AmplabJenkins · 2015-05-01T18:57:19Z

Merged build started.

SparkQA · 2015-05-01T18:57:57Z

Test build #31584 has started for PR 5575 at commit ca1151d.

SparkQA · 2015-05-01T20:36:12Z

Test build #31584 has finished for PR 5575 at commit ca1151d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- final class Binarizer extends Transformer with HasInputCol with HasOutputCol

AmplabJenkins · 2015-05-01T20:36:16Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-01T20:36:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31584/
Test PASSed.

Currently we're doing port retries in the TransportServer level, but this is not specified by the TransportContext API and it has other further-reaching impacts like causing undesirable behvior for the Yarn and Standalone shuffle services.

aarondav · 2015-05-08T18:28:17Z

cc @andrewor14 @pwendell We should consider merging this into the 1.4 branch in case there is another RC. It has been an outstanding issue for a while.

AmplabJenkins · 2015-05-08T18:32:12Z

Merged build triggered.

AmplabJenkins · 2015-05-08T18:32:22Z

Merged build started.

SparkQA · 2015-05-08T18:32:50Z

Test build #32254 has started for PR 5575 at commit 3c2d6ed.

SparkQA · 2015-05-08T20:15:19Z

Test build #32254 has finished for PR 5575 at commit 3c2d6ed.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class EnumUtil
- class Binarizer(JavaTransformer, HasInputCol, HasOutputCol):
- class IDF(JavaEstimator, HasInputCol, HasOutputCol):
- class IDFModel(JavaModel):
- class Normalizer(JavaTransformer, HasInputCol, HasOutputCol):
- class OneHotEncoder(JavaTransformer, HasInputCol, HasOutputCol):
- class PolynomialExpansion(JavaTransformer, HasInputCol, HasOutputCol):
- class RegexTokenizer(JavaTransformer, HasInputCol, HasOutputCol):
- class StandardScaler(JavaEstimator, HasInputCol, HasOutputCol):
- class StandardScalerModel(JavaModel):
- class StringIndexer(JavaEstimator, HasInputCol, HasOutputCol):
- class StringIndexerModel(JavaModel):
- class Tokenizer(JavaTransformer, HasInputCol, HasOutputCol):
- class VectorIndexer(JavaEstimator, HasInputCol, HasOutputCol):
- class Word2Vec(JavaEstimator, HasStepSize, HasMaxIter, HasSeed, HasInputCol, HasOutputCol):
- class Word2VecModel(JavaModel):
- class HasSeed(Params):
- class HasTol(Params):
- class HasStepSize(Params):

AmplabJenkins · 2015-05-08T20:15:24Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-08T20:15:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32254/
Test PASSed.

andrewor14 · 2015-05-09T00:13:17Z

Nice, I merge.

andrewor14 · 2015-05-09T00:13:21Z

master 1.4

Currently we're doing port retries in the TransportServer level, but this is not specified by the TransportContext API and it has other further-reaching impacts like causing undesirable behavior for the Yarn and Standalone shuffle services. Author: Aaron Davidson <aaron@databricks.com> Closes #5575 from aarondav/port-bind and squashes the following commits: 3c2d6ed [Aaron Davidson] Oops, never do it. a5d9432 [Aaron Davidson] Remove shouldHostShuffleServiceIfEnabled e901eb2 [Aaron Davidson] fix local-cluster mode for ExternalShuffleServiceSuite 59e5e38 [Aaron Davidson] [SPARK-6955] Perform port retries at NettyBlockTransferService level (cherry picked from commit ffdc40c) Signed-off-by: Andrew Or <andrew@databricks.com>

Currently we're doing port retries in the TransportServer level, but this is not specified by the TransportContext API and it has other further-reaching impacts like causing undesirable behavior for the Yarn and Standalone shuffle services. Author: Aaron Davidson <aaron@databricks.com> Closes apache#5575 from aarondav/port-bind and squashes the following commits: 3c2d6ed [Aaron Davidson] Oops, never do it. a5d9432 [Aaron Davidson] Remove shouldHostShuffleServiceIfEnabled e901eb2 [Aaron Davidson] fix local-cluster mode for ExternalShuffleServiceSuite 59e5e38 [Aaron Davidson] [SPARK-6955] Perform port retries at NettyBlockTransferService level

aarondav force-pushed the port-bind branch from c671f6f to a3dc42d Compare April 18, 2015 17:59

aarondav mentioned this pull request Apr 18, 2015

[SPARK-6955][NETWORK]Do not let Yarn Shuffle Server retry its server port. #5537

Closed

vanzin reviewed Apr 20, 2015
View reviewed changes

andrewor14 reviewed Apr 21, 2015
View reviewed changes

srowen mentioned this pull request Apr 25, 2015

[SPARK-7086][Deploy]Do not retry when public service start on port #5657

Closed

aarondav force-pushed the port-bind branch from df60b80 to d38075b Compare May 1, 2015 16:00

aarondav added 3 commits May 8, 2015 11:27

fix local-cluster mode for ExternalShuffleServiceSuite

e901eb2

Remove shouldHostShuffleServiceIfEnabled

a5d9432

aarondav force-pushed the port-bind branch from ca1151d to 4813b6f Compare May 8, 2015 18:27

Oops, never do it.

3c2d6ed

aarondav force-pushed the port-bind branch from 4813b6f to 3c2d6ed Compare May 8, 2015 18:31

asfgit closed this in ffdc40c May 9, 2015

[SPARK-6955] Perform port retries at NettyBlockTransferService level #5575

[SPARK-6955] Perform port retries at NettyBlockTransferService level #5575

Conversation

aarondav commented Apr 18, 2015

SparkQA commented Apr 18, 2015

SparkQA commented Apr 18, 2015

AmplabJenkins commented Apr 18, 2015

aarondav commented Apr 18, 2015

aarondav commented Apr 18, 2015

SparkQA commented Apr 18, 2015

SparkQA commented Apr 18, 2015

AmplabJenkins commented Apr 18, 2015

aarondav commented Apr 19, 2015

SparkQA commented Apr 19, 2015

SparkQA commented Apr 19, 2015

AmplabJenkins commented Apr 19, 2015

aarondav commented Apr 19, 2015

SparkQA commented Apr 19, 2015

SparkQA commented Apr 19, 2015

AmplabJenkins commented Apr 19, 2015

SparkQA commented Apr 19, 2015

SparkQA commented Apr 20, 2015

AmplabJenkins commented Apr 20, 2015

SaintBacchus commented Apr 20, 2015

vanzin Apr 20, 2015

Choose a reason for hiding this comment

aarondav Apr 20, 2015

Choose a reason for hiding this comment

vanzin commented Apr 20, 2015

andrewor14 Apr 21, 2015

Choose a reason for hiding this comment

SparkQA commented Apr 27, 2015

AmplabJenkins commented May 1, 2015

SparkQA commented May 1, 2015

AmplabJenkins commented May 1, 2015

AmplabJenkins commented May 1, 2015

AmplabJenkins commented May 1, 2015

AmplabJenkins commented May 1, 2015

SparkQA commented May 1, 2015

SparkQA commented May 1, 2015

AmplabJenkins commented May 1, 2015

AmplabJenkins commented May 1, 2015

aarondav commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

andrewor14 commented May 9, 2015

andrewor14 commented May 9, 2015