Build a Continuous Security Scanning Pipeline with Jenkins and Kubernetes

Introduction

Initial public release at BsidesLv 2018:

Video: https://www.youtube.com/watch?v=Lxk2N4dqVnI&feature=youtu.be&t=32094

Presentation slides: bsideslv_19_prensentation_vf.pdf

This guide will take you through the steps necessary to build a security automation framework by converting security tooling into micro services, and using Kubernetes as an orchestration engine in conjunction with your favorite CI/CD tool.

The framework also includes an easy to deploy AppSec SIEM for tracking/monitoring the continuous scanning activity along with the development lifecycle.

For the sake of demonstration, this guidline will us Google Container Engine and Jenkins to orchestrate the security scanning activity.

If you are not familiar with basic Kubernetes concepts, have a look at Kubernetes 101.

In order to accomplish this goal you will use the following Jenkins plugins:

• Jenkins Kubernetes Plugin - start Jenkins build executor containers in the Kubernetes cluster when builds are requested, terminate those containers when builds complete, freeing resources up for the rest of the cluster
• Jenkins Pipelines - define our build pipeline declaratively and keep it checked into source code management alongside our application code
• GitHub Groovy Libraries - Allows pipeline Groovy libraries to be loaded on the fly from GitHub.

Prerequisites

1. A Google Cloud Platform Account
2. Enable the Compute Engine, Container Engine, and Container Builder APIs

Kubernetes and Jenkins Setup

Clone the lab repository in your cloud shell using the following link :

Create a Kubernetes Cluster

We will use Google Container Engine to create and manage your Kubernetes cluster. Provision the cluster with gcloud:

gcloud container clusters create jenkins-cd \
--num-nodes 2 \
--machine-type n1-standard-2 \
--scopes "https://www.googleapis.com/auth/projecthosting,cloud-platform"

Once that operation completes download the credentials for your cluster using the gcloud CLI:

gcloud container clusters get-credentials jenkins-cd
Fetching cluster endpoint and auth data.
kubeconfig entry generated for jenkins-cd.

Confirm that the cluster is running and kubectl is working by listing pods:

kubectl get pods
No resources found.

You should see No resources found..

Configure and Install Jenkins

We will use a custom values file to add the GCP specific plugin necessary to use service account credentials to reach your Cloud Source Repository.

1. Use the Helm CLI to deploy the chart with your configuration set.

    helm install -n cd stable/jenkins -f jenkins/values.yaml --wait

1. Once that command completes ensure the Jenkins pod goes to the Running state and the container is in the READY state:

    kubectl get pods
    NAME                          READY     STATUS    RESTARTS   AGE
    cd-jenkins-7c786475dd-vbhg4   1/1       Running   0          1m

2. Configure the Jenkins service account to be able to deploy to the cluster.

    kubectl create clusterrolebinding jenkins-deploy --clusterrole=cluster-admin --serviceaccount=default:cd-jenkins
    clusterrolebinding.rbac.authorization.k8s.io/jenkins-deploy created

3. Run the following command to setup port forwarding to the Jenkins UI from the Cloud Shell

    export POD_NAME=$(kubectl get pods -l "component=cd-jenkins-master" -o jsonpath="{.items[0].metadata.name}")
    kubectl port-forward $POD_NAME 8080:8080 >> /dev/null &

4. Create a persitance volume to share scanning results between containers and the ELK stack.

    kubectl create -f jenkins/pv.yaml

Connect to Jenkins

1. The Jenkins chart will automatically create an admin password for you. To retrieve it, run:

    printf $(kubectl get secret cd-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo

2. To get to the Jenkins user interface, click on the Web Preview button in cloud shell, then click "Preview on port 8080" :

You should now be able to log in with username admin and your auto generated password.

Setup Defect Dojo as vulnerability management system (Optional) :

If your security scanner doesn't include a vulnerability management system, or you are planning to use multiple tools, you can use DefectDojo as a vulnerability correlation system.

Once installed, generate an API token and configure it as a secret into Kubernetes (or Jenkins) in order to use it later to automatically gather results after each security scan.

Convert your security scanners to a docker image

This repo provides two examples of security scanners :

• OWASP ZAP for Dynamic Analysis (open source)
• Checkmarx for Static Analysis

Note: you can always use the official/public docker image of your favorite scanner.

To build the docker image of OWASP ZAP :

    cd scanners/dast/zap
    docker build -f Dockerfile-stable -t yourcompany/zap .

Each Dockerfile is orgnized as follow :

• Setup & configuration of the security testing tool
• Pushing metrics to DefectDojo (optionnal)
• Storing reports and scanning logs to the shared volume

    cd scanners/dast/zap
    docker build -f Dockerfile-stable -t yourcompany/zap .

Then push the image to your internal/external Docker registry :

    docker push yourcompany/zap

A Jenkinsfile is also provided in order to automate the build process of your docker image.

Define a groovy function for your security scanner

In order to make it more flexible and easier to run security analysis from an existing (or a seperated) DevOps pipeline, a groovy function can be also defined in order to provide a high level definition of the scanning process.

Inject the security scanning step into an exsiting pipeline

As an example, we can use the groovy function we defined for OWASP ZAP in order to run a dynamic analysis :

@Library('github.com/TmmmmmR/SecLib@master') _

owaspzap{
    engagement_id      = "cicd"
    target_url         = "https://example.com"
}

Setup ELK for security analytics

There are a few configuration steps for setting up the ELK stack :

• Configure logstash to pull the scanning reports and logs
• Define and configure a logstash pipeline to manipulate data issued from each scanners
• Configure Kibana as a presentation layer

This repo include a docker-compose file to quickly/automaticly setup an ELK stack with some testing reports and dashboards.

The configuration files of this repo can be easily imported/reused into an existing ELK environnement.

Extened the framework

This framework can be extended to support other security testing tools by following the steps described bellow :

• Convert your secutiy testing tools into a [docker image](add link)
• Define a groovy function to
• Configure a Logstash pipeline (example for zap) to gather the scanning logs/reports and ship them into Elasticsearch
• Define your own visualization/dasboard to view the scanning results, or use the provided example.

References & previous work

• ELK Stack
• VulnWhisperer
• Building A Security Test Automation Framework by Riccardo Ten Cate
• secureCodeBox
• AppSecPipeline