SPARK-1974. Most examples fail at startup because spark.master is not set

[srowen]

Most example code has a few lines like:

val sparkConf = new SparkConf().setAppName("Foo")
val sc = new SparkContext(sparkConf)

The SparkContext constructor throws a SparkException if spark.master is not set though, so this fails immediately.

This changes all examples to call new SparkContext("local[2]", "Foo") or similar. local[2] because it's necessary for streaming examples and because it's otherwise used as the default over local[1] in Spark.

(Since this started off when debugging the Kafka streaming code, I also included some refinements there to the logging and resource management. Lightly related.)

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[vanzin]

Won't this break things if you try to submit the examples with spark-submit (which I think is the New And Approved Way (tm))?

spark-submit will set spark.master, and if I'm following things correctly, this change will override that.

[srowen]

Hm good question, you could make it work with -Dspark.master=.... This would overwrite those settings. It seems like the examples are intended to work without that setting though. Better to make the global internal default something like local[2]?

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15309/

[vanzin]

If you really want to keep the tests working outside of spark-submit, I'd suggest using the SparkContext(SparkConf) constructor instead, and using SparkConf.setIfMissing() to set "spark.master".

Given that bin/run-example uses spark-submit, I'm not sure we should bother though.

[srowen]

Yeah I think it's essential to not prevent -Dspark.master=... from working, oops. I think it may be useful to have this work if one copies-and-pastes too, as I just did. The javadoc doesn't indicate that you have to set the master either. I will rework it to use setIfMissing()

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[srowen]

I pushed again, with setIfMissing. Is it better in the SparkConf constructor? or I am off base here?

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15321/

[pwendell]

Hey Sean, how are you running the examples. Are you using the run-example script? That script should set the master to local[*] if the user hasn't specified it, which will use all cores locally. I think in some cases we might need to update the javadocs in examples to tell users to use run-example.

[srowen]

I'm just copying-and-pasting to get something similar running externally. Maybe it's a little surprising that the example code doesn't work that way -- being in a main() kind of suggests this is a stand-alone program. Maybe just me.

I think there are a few possibilities:

• Change all example code to set master if missing (that's what the current PR does)
• Change SparkConf to do something similar as a global default
• Just update javadoc to make it clear that the examples require the spark.master system property to be set

I slightly prefer one of the first two on the principle of least surprise, but can go any direction. I think at least the third should be done. What say everyone?

[vanzin]

Hey Sean,

I'm still a little confused about what it is you're doing. What is the javadoc you refer to? I've looked at a few classes in org.apache.spark.examples and I don't really see a lot of comments.

If you're using spark-submit or run-example, you shouldn't be running into this issue ("spark.master" not set) at all. You'd only run into it if using spark-class directly.

In 0.9 the examples (or at least a few of them) used to take the master as the first argument. In 1.0.0 it seems the approach taken was to just ignore that argument since it's provided automatically by spark-submit. Your command line is "backwards compatible" although that one argument is just ignored. I don't know how important that is for examples, though - I'd rather have the examples as an example of how to write an app, and having that old argument there kinda defeats that purpose.

So, if there are docs telling people to run the examples directly with java or spark-class, we should fix those to use run-example / spark-submit instead.

Perhaps a future approach could be to have a "SparkApp" base class that calls a "run(SparkContext)" method so that initializing the context clearly becomes the job of the framework. But that approach has all sorts of other issues.

[syedhashmi]

@vanzin : I was running into this issue while trying to run and debug examples from intellij. It was working fine pre-1.0.

[vanzin]

Ah, that case makes sense. For that I think Sean's current fix should be enough. But still, if there's documentation telling users to run examples that way, it should probably be fixed.

[srowen]

@vanzin I was just copying-and-pasting the substance of the main() method from an example to modify it. You are right that most examples tell you to use run-example and that works. One asks you set a master arg directly but it's just one.

Nothing wrong with that per se, but the code almost completely configures Spark, except for the master. At a glance, I thought it wasn't intentional, and that spark.master was supposed to default to local[2].

I don't quite like my change -- a global default is much simpler but has perhaps more implications. I am not really wedded to inserting a default.

I could instead make sure every bit of example code notes in javadoc that run-example should be used. (Really this all started because I wanted to also suggest a few touch-ups to KafkaInputDStream too so wouldn't mind slipping that in -- that's how I got started...)

[mateiz]

I don't think we want to change the examples like this. Instead, we should make the default spark.master be local instead of having an exception (I believe there was a JIRA for that). Nonetheless the examples show how you are supposed to write a spark-submit app today so we should leave them as they are.

[srowen]

Got it. I'm going to close this, and will open a PR for the JIRA you may have in mind (https://issues.apache.org/jira/browse/SPARK-1906) to just set a spark.master default. (I'll deal with this Kafka change separately). Then it's there for consideration.