- Background
- Overview
- Requirements
- Example
- Generator Options
- Development and Testing
- Argo CD Integration 🤖
At Viaduct, we manage our Kubernetes resources via the GitOps pattern; however, we could not find a solution compatible with our stack for managing secrets via the GitOps paradigm. We built KSOPS
to connect kustomize to SOPS and integrated it with Argo CD to safely manage our secrets the same way we manage the rest our Kubernetes manifest.
KSOPS
, or kustomize-SOPS, is a kustomize plugin for SOPS encrypted resources. KSOPS
can be used to decrypt any Kubernetes resource, but is most commonly used to decrypt encrypted Kubernetes Secrets and ConfigMaps. As a kustomize plugin, KSOPS
allows you to manage, build, and apply encrypted manifests the same way you manage the rest of your Kubernetes manifests.
- Go
- kustomize built with Go (See details below)
- SOPS
- gpg
Before continuing, verify your installation of Go, SOPS, and gpg
. Below are a few non-comprehensive commands to quickly check your installations:
# Verify Go is installed and your $GOPATH is set
go env
# Verify SOPS is installed
sops --version
# Verify gpg is installed
gpg --help
# export GO111MODULE=on
go get -u github.com/viaduct-ai/kustomize-sops
# cd into the root directory
cd $GOPATH/src/github.com/viaduct-ai/kustomize-sops
# KSOPS is built with latest kustomize
# If you want to change versions, update the installation script with your desired version and make sure to check that the KSOPS tests still pass
# If you want to change versions below kustomize v3.3.0, use the KSOPS v1.0 or go-1.12 release!
make kustomize
# Don't forget to define XDG_CONFIG_HOME in your .bashrc/.zshrc
echo "export XDG_CONFIG_HOME=\$HOME/.config" >> $HOME/.bashrc
source $HOME/.bashrc
make install
To simplify local development and testing, we use PGP test keys. To import the keys, run the following command:
make import-test-keys
If you are following this tutorial, be sure to run this before the following steps. The PGP keys will also be imported when you run make test
See SOPS for details.
For this example and testing, KSOPS
relies on the SOPS
creation rules defined in .sops.yaml
. To make encrypted secrets more readable, we suggest using the following encryption regex to only encrypt data
and stringData
values. This leaves non-sensitive fields, like the secret's name, unencrypted and human readable.
Note: You only have to modify .sops.yaml
if you want to use your key management service in this example instead of the default PGP key imported in the previous step.
creation_rules:
- encrypted_regex: '^(data|stringData)$'
# Specify kms/pgp/etc encryption key
# This tutorial uses a local PGP key for encryption.
# DO NOT USE IN PRODUCTION ENV
pgp: 'FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4'
# Optionally you can configure to use a providers key store
# kms: XXXXXX
# gcp_kms: XXXXXX
# Create a local Kubernetes Secret
cat <<EOF > secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: YWRtaW4=
password: MWYyZDFlMmU2N2Rm
EOF
# Encrypt with SOPS CLI
# Specify SOPS configuration in .sops.yaml
sops -e secret.yaml > secret.enc.yaml
# Create a local Kubernetes Secret
cat <<EOF > secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
# Specify a name
name: example-secret-generator
files:
- ./secret.enc.yaml
EOF
cat <<EOF > kustomization.yaml
generators:
- ./secret-generator.yaml
EOF
# Build with kustomize to verify
kustomize build --enable_alpha_plugins .
Detailed example of kustomize Go plugin
- Validate
ksops.so
is in thekustomize
plugin path$XDG_CONFIG_HOME/kustomize/plugin/viaduct.ai/v1/ksops/ksops.so
- Check your
kustomize
executable was built by Gowhich kustomize
kustomize version
- Check the Go version in
go.mod
matches the Go version used to buildkustomize
- Check the
kustomize
version specified ingo.mod
matches the installed version ofkustomize
kustomize version
Someone might have already encountered your issue.
https://github.com/viaduct-ai/kustomize-sops/issues
KSOPS
supports the same annotation based generator options as kustomize exec plugins. The supported annotations are:
kustomize.config.k8s.io/needs-hash
kustomize.config.k8s.io/behavior
For information, read the kustomize generator options documentation.
Sometimes there is a default secret as part of a project's base manifests, like the base Argo CD secret, that you want to replace
in your overlay. Other times, you have parts of base secret that are common across different overlays but you want to partially update, or merge
, changes specific to each overlay as well. You can achieve both of these goals by simply adding the following annotations to your encrypted secrets:
apiVersion: v1
kind: Secret
metadata:
name: argocd-secret
annotations:
# replace the base secret data/stringData values with these encrypted data/stringData values
kustomize.config.k8s.io/behavior: replace
type: Opaque
data:
# Encrypted data here
stringData:
# Encrypted data here
apiVersion: v1
kind: Secret
metadata:
name: argocd-secret
annotations:
# merge the base secret data/stringData values with these encrypted data/stringData values
kustomize.config.k8s.io/behavior: merge
type: Opaque
data:
# Encrypted data here
stringData:
# Encrypted data here
Before developing or testing KSOPS
, ensure all external requirements are properly installed.
# Setup development environment
make setup
KSOPS
implements the kustomize plugin API in ksops.go
.
KSOPS
's logic is intentionally simple. Given a list of SOPS encrypted Kubernetes manifests, it iterates over each file and decrypts it via SOPS decrypt library. KSOPS
assumes nothing about the structure of the encrypted resource and relies on kustomize for manifest validation. KSOPS
expects the encryption key to be accessible. This is important to consider when using KSOPS
for CI/CD.
Testing KSOPS
requires:
- Configuring a encryption key and other
SOPS
configuration in.sops.yaml
- Building
KSOPS
as a Go plugin - Copying the Go plugin to the kustomize plugin path
- Generating encrypted test files
- Running the Go tests
Everything but setting up .sops.yaml
is handle for you by make test
. After defining .sops.yaml
, test KSOPS
running the following command from the repo's root directory:
make test
KSOPS
becomes even more powerful when integrated with a CI/CD pipeline. By combining KSOPS
with Argo CD, you can manage Kubernetes secrets via the same Git Ops pattern you use to manage the rest of your kubernetes manifests. To integrate KSOPS
and Argo CD, you will need to update the Argo CD ConifgMap and create a strategic merge patch or a custom Argo CD build. Don't forget to inject any necessary credentials (i.e AWS credentials) when deploying the Argo CD + KSOPS
build!
As of now to allow Argo CD to use kustomize plugins you must use the enable_alpha_plugins
flag. This is configured by the kustomize.buildOptions
setting in the Argo CD ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
labels:
app.kubernetes.io/name: argocd-cm
app.kubernetes.io/part-of: argocd
data:
kustomize.buildOptions: "--enable_alpha_plugins"
The simplest way to integrate KSOPS
with Argo CD is with a strategic merge patch on the Argo CD repo server deployment. The patch below uses an init container to build KSOPS
and kustomize and volume mount to inject the KSOPS
plugin and override the kustomize executable.
# argo-cd-repo-server-ksops-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
template:
spec:
# 1. Define an emptyDir volume which will hold the custom binaries
volumes:
- name: custom-tools
emptyDir: {}
# 2. Use an init container to download/copy custom binaries into the emptyDir
initContainers:
- name: install-ksops
# Match Argo CD Go version
image: viaductoss/ksops:v2.1.2-go-1.14
command: ["/bin/sh", "-c"]
args:
- echo "Installing KSOPS...";
export PKG_NAME=ksops;
mv ${PKG_NAME}.so /custom-tools/;
mv $GOPATH/bin/kustomize /custom-tools/;
echo "Done.";
volumeMounts:
- mountPath: /custom-tools
name: custom-tools
# 3. Volume mount the custom binary to the bin directory (overriding the existing version)
containers:
- name: argocd-repo-server
volumeMounts:
- mountPath: /usr/local/bin/kustomize
name: custom-tools
subPath: kustomize
# Verify this matches a XDG_CONFIG_HOME=/.config env variable
- mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops.so
name: custom-tools
subPath: ksops.so
# 4. Set the XDG_CONFIG_HOME env variable to allow kustomize to detect the plugin
env:
- name: XDG_CONFIG_HOME
value: /.config
## If you use AWS or GCP KMS, don't forget to include the necessary credentials to decrypt the secrets!
# - name: AWS_ACCESS_KEY_ID
# valueFrom:
# secretKeyRef:
# name: argocd-aws-credentials
# key: accesskey
# - name: AWS_SECRET_ACCESS_KEY
# valueFrom:
# secretKeyRef:
# name: argocd-aws-credentials
# key: secretkey
Alternatively, for more control and faster pod start times you can build a custom docker image.
ARG ARGO_CD_VERSION="v1.5.5"
# Always match Argo CD Dockerfile's Go version!
# https://github.com/argoproj/argo-cd/blob/master/Dockerfile
ARG KSOPS_VERSION="v2.1.2-go-1.14"
#--------------------------------------------#
#--------Build KSOPS and Kustomize-----------#
#--------------------------------------------#
FROM viaductoss/ksops:$KSOPS_VERSION as ksops-builder
#--------------------------------------------#
#--------Build Custom Argo Image-------------#
#--------------------------------------------#
FROM argoproj/argocd:$ARGO_CD_VERSION
# Switch to root for the ability to perform install
USER root
# Set the kustomize home directory
ENV XDG_CONFIG_HOME=$HOME/.config
ENV KUSTOMIZE_PLUGIN_PATH=$XDG_CONFIG_HOME/kustomize/plugin/
ARG PKG_NAME=ksops
# Override the default kustomize executable with the Go built version
COPY --from=ksops-builder /go/bin/kustomize /usr/local/bin/kustomize
# Copy the plugin to kustomize plugin path
COPY --from=ksops-builder /go/src/github.com/viaduct-ai/kustomize-sops/* $KUSTOMIZE_PLUGIN_PATH/viaduct.ai/v1/${PKG_NAME}/
# Switch back to non-root user
USER argocd