Skip to content

Commit

Permalink
Merge branch 'master' into rw_beam_zetasql
Browse files Browse the repository at this point in the history
  • Loading branch information
amaliujia committed Aug 12, 2019
2 parents 46c9f24 + 6c65527 commit 5adf530
Show file tree
Hide file tree
Showing 404 changed files with 17,393 additions and 4,225 deletions.
1 change: 1 addition & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
Go | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
Python | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | ---

Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion .test-infra/jenkins/CommonJobProperties.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ class CommonJobProperties {
}
branch('${sha1}')
extensions {
cleanAfterCheckout()
wipeOutWorkspace()
relativeTargetDirectory(checkoutDir)
if (!allowRemotePoll) {
disableRemotePoll()
Expand Down
9 changes: 9 additions & 0 deletions .test-infra/jenkins/Infrastructure.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,15 @@ class Infrastructure {
}
}

static void scaleCluster(def context, String jobName, Integer workerCount) {
context.steps {
// Keep one extra Dataproc VM for Flink's Job Manager
workerCount += 1
shell("echo Changing number of workers to ${workerCount}")
shell("gcloud dataproc clusters update ${getClusterName(jobName)} --num-workers=${workerCount} --quiet")
}
}

private static GString getClusterName(String jobName) {
return "${jobName.toLowerCase().replace("_", "-")}-\$BUILD_ID"
}
Expand Down
108 changes: 108 additions & 0 deletions .test-infra/jenkins/Kubernetes.groovy
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
/** Facilitates creation of jenkins steps to setup and cleanup Kubernetes infrastructure. */
class Kubernetes {

private static final String KUBERNETES_DIR = '"$WORKSPACE/src/.test-infra/kubernetes"'

private static final String KUBERNETES_SCRIPT = "${KUBERNETES_DIR}/kubernetes.sh"

private static def job

private static String kubeconfigLocation

private static String namespace

private Kubernetes(job, String kubeconfigLocation, String namespace) {
this.job = job
this.kubeconfigLocation = kubeconfigLocation
this.namespace = namespace
}

/**
* Creates separate kubeconfig, kubernetes namespace and specifies related cleanup steps.
*
* @param job - jenkins job
* @param kubeconfigLocation - place where kubeconfig will be created
* @param namepsace - kubernetes namespace
*/
static Kubernetes create(job, String kubeconfigLocation, String namespace) {
Kubernetes kubernetes = new Kubernetes(job, kubeconfigLocation, namespace)
setupKubeconfig()
setupNamespace()
addCleanupSteps()
return kubernetes
}

private static void setupKubeconfig() {
job.steps {
shell("cp /home/jenkins/.kube/config ${kubeconfigLocation}")
environmentVariables {
env('KUBECONFIG', kubeconfigLocation)
}
}
}

private static void setupNamespace() {
job.steps {
shell("${KUBERNETES_SCRIPT} createNamespace ${namespace}")
environmentVariables {
env('KUBERNETES_NAMESPACE', namespace)
}
}
}

private static void addCleanupSteps() {
job.publishers {
postBuildScripts {
steps {
shell("${KUBERNETES_SCRIPT} deleteNamespace ${namespace}")
shell("rm ${kubeconfigLocation}")
}
onlyIfBuildSucceeds(false)
onlyIfBuildFails(false)
}
}
}

/**
* Specifies steps to run Kubernetes .yaml script.
*/
void apply(String pathToScript) {
job.steps {
shell("${KUBERNETES_SCRIPT} apply ${pathToScript}")
}
}

/**
* Specifies steps that will save specified load balancer serivce address
* as an environment variable that can be used in later steps if needed.
*
* @param serviceName - name of the load balancer Kubernetes service
* @param referenceName - name of the environment variable
*/
void loadBalancerIP(String serviceName, String referenceName) {
job.steps {
String command = "${KUBERNETES_SCRIPT} loadBalancerIP ${serviceName}"
shell("set -eo pipefail; eval ${command} | sed 's/^/${referenceName}=/' > job.properties")
environmentVariables {
propertiesFile('job.properties')
}
}
}
}
23 changes: 23 additions & 0 deletions .test-infra/jenkins/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,29 @@ Beam Jenkins overview page: [link](https://builds.apache.org/view/A-D/view/Beam/
| beam_PerformanceTests_WordCountIT_Py37 | [cron](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/) | `Run Python37 WordCountIT Performance Test` | [![Build Status](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/badge/icon)](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37) |
| beam_PerformanceTests_XmlIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_XmlIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_XmlIOIT_HDFS/) | `Run Java XmlIO Performance Test` | [![Build Status](https://builds.apache.org/job/beam_PerformanceTests_XmlIOIT/badge/icon)](https://builds.apache.org/job/beam_PerformanceTests_XmlIOIT) [![Build Status](https://builds.apache.org/job/beam_PerformanceTests_XmlIOIT_HDFS/badge/icon)](https://builds.apache.org/job/beam_PerformanceTests_XmlIOIT_HDFS) |

### Load test Jobs

| Name | Link | PR Trigger Phrase | Cron Status |
|------|------|-------------------|-------------|
| beam_LoadTests_Python_coGBK_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch_PR/) | Run Load Tests Python CoGBK Flink Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch/) |
| beam_LoadTests_Java_CoGBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch_PR/) | Run Load Tests Java CoGBK Dataflow Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch/) |
| beam_LoadTests_Java_CoGBK_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming_PR/) | Run Load Tests Java CoGBK Dataflow Streaming | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming/) |
| beam_LoadTests_Python_CoGBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch_PR/) | Run Load Tests Python CoGBK Dataflow Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch/) |
| beam_LoadTests_Python_Combine_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch_PR/) | Run Load Tests Python Combine Flink Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/) |
| beam_LoadTests_Java_Combine_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch_PR/) | Run Load Tests Java Combine Dataflow Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch/) |
| beam_LoadTests_Java_Combine_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming_PR/) | Run Load Tests Java Combine Dataflow Streaming | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming/) |
| beam_LoadTests_Python_Combine_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch_PR/) | Run Python Load Tests Combine Dataflow Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/) |
| beam_LoadTests_Python_GBK_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch_PR/) | Run Load Tests Python GBK Flink Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/) |
| beam_LoadTests_Java_GBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch_PR/) | Run Load Tests Java GBK Dataflow Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/) |
| beam_LoadTests_Java_GBK_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming_PR/) | Run Load Tests Java GBK Dataflow Streaming | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/) |
| beam_LoadTests_Python_GBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch_PR/) | Run Load Tests Python GBK Dataflow Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch/) |
| beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch_PR/) | Run Load Tests Python GBK reiterate Dataflow Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/) |
| beam_Java_LoadTests_Smoke | [phrase](https://builds.apache.org/job/beam_Java_LoadTests_Smoke_PR/) | Run Java Load Tests Smoke | |
| beam_Python_LoadTests_Smoke | [phrase](https://builds.apache.org/job/beam_Python_LoadTests_Smoke_PR/) | Run Python Load Tests Smoke | |
| beam_LoadTests_Java_ParDo_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch_PR/) | Run Load Tests Java ParDo Dataflow Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch/) |
| beam_LoadTests_Java_ParDo_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming_PR/) | Run Load Tests Java ParDo Dataflow Streaming | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming/) |
| beam_LoadTests_Python_ParDo_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch_PR/) | Run Python Load Tests ParDo Dataflow Batch | [![Build Status](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch/badge/icon)](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch/) |

### Inventory Jobs

| Name | Link | PR Trigger Phrase | Cron Status |
Expand Down
Loading

0 comments on commit 5adf530

Please sign in to comment.