forked from apache/spark
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request apache#25 from mesosphere/spark-new-config
[SPARK-225] new config
- Loading branch information
Showing
13 changed files
with
206 additions
and
271 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
docker: | ||
bin/make-docker.sh | ||
|
||
package: | ||
bin/make-package.py | ||
|
||
universe: | ||
bin/make-universe.sh | ||
|
||
test: | ||
bin/test.sh | ||
|
||
.PHONY: package docker universe test |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,62 +1,47 @@ | ||
Spark DCOS Package | ||
=== | ||
# Spark DCOS Package | ||
|
||
This repo lets you configure, build, and test a new Spark DCOS package. | ||
It is the source for the Spark package in universe. If you wish to modify | ||
that package, you should do so here, and generate a new package as | ||
described below. | ||
|
||
Configuring | ||
--- | ||
## Configuring | ||
|
||
edit `manifest.json`. | ||
|
||
Create a package | ||
--- | ||
## Push a docker image | ||
|
||
This will make a docker image from the distribution specified in `manifest.json` | ||
|
||
``` | ||
export DOCKER_IMAGE=... | ||
./bin/make-package.sh | ||
DOCKER_IMAGE=<name> make docker | ||
``` | ||
|
||
This command builds a docker image, pushes it, and writes a new | ||
package to `build/package`. It uses the components listed in | ||
`manifest.json`. | ||
## Create a package | ||
|
||
Create a universe | ||
--- | ||
Write a package to `build/package`. Use the `DOCKER_IMAGE` name you | ||
created above. | ||
|
||
``` | ||
./bin/make-universe.sh | ||
DOCKER_IMAGE=<name> make package | ||
``` | ||
|
||
This produces a new universe in `build/universe`. You can then point your | ||
local `dcos` to this location via `dcos config set package.sources`. | ||
## Create a universe | ||
|
||
Create a docker image | ||
--- | ||
Write a universe to `build/universe`. You can then upload this to | ||
e.g. S3, and point your DCOS cluster at it via `dcos package repo | ||
add`. | ||
|
||
``` | ||
./bin/make-docker.sh <spark-dist> <image> | ||
make universe | ||
``` | ||
|
||
* `<spark-dist>`: path to spark distribution | ||
* `<image>`: name of docker image | ||
|
||
This creates a new docker image from the given spark distribution. It | ||
is called by `./bin/make-package.sh`. | ||
|
||
|
||
Test | ||
--- | ||
## Test | ||
|
||
``` | ||
./bin/test.sh | ||
``` | ||
|
||
This performs every build step, including tests. It builds spark, the docker image, | ||
the package, and the universe. It spins up a CCM cluster and tests spark against that | ||
cluster. | ||
|
||
It requires several env variables. Read the comment at the top of the file for a | ||
complete description. | ||
This requires several env variables, and is primarily used in CI. | ||
Read the comment at the top of the file for a complete description. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,14 +1,35 @@ | ||
#!/usr/bin/env bash | ||
|
||
# Usage: | ||
# ./bin/make-docker.sh <spark-dist-dir> <image> | ||
set -e -o pipefail | ||
|
||
# ENV vars: | ||
# DOCKER_IMAGE - <image>:<version> | ||
# SPARK_DIST_URI (optional) - e.g. http://<domain>/spark-1.2.3.tgz | ||
|
||
if [ -z "${SPARK_DIST_URI}" ]; then | ||
SPARK_URI=$(cat manifest.json | jq .spark_uri) | ||
SPARK_URI="${SPARK_URI%\"}" | ||
SPARK_URI="${SPARK_URI#\"}" | ||
SPARK_DIST_URI=${SPARK_URI} | ||
fi | ||
|
||
DIST_TGZ=$(basename "${SPARK_DIST_URI}") | ||
DIST="${DIST_TGZ%.*}" | ||
|
||
# fetch spark | ||
mkdir -p build/dist | ||
[ -f "build/dist/${DIST_TGZ}" ] || curl -o "build/dist/${DIST_TGZ}" "${SPARK_DIST_URI}" | ||
tar xvf build/dist/spark*.tgz -C build/dist | ||
|
||
# create docker context | ||
rm -rf build/docker | ||
mkdir -p build/docker/dist | ||
cp -r "$1/." build/docker/dist | ||
cp -r "build/dist/${DIST}/." build/docker/dist | ||
cp -r conf/* build/docker/dist/conf | ||
cp -r docker/* build/docker | ||
|
||
pushd build/docker | ||
docker build -t $2 . | ||
popd | ||
# build docker | ||
(cd build/docker && docker build -t "${DOCKER_IMAGE}" .) | ||
|
||
# push docker | ||
docker push "${DOCKER_IMAGE}" |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.