Skip to content

krouma/gpu-mutating-webhook

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GPU Mutating Webhook

This webhook fixes the problem that no GPU request container can see all GPUs.

Prerequisites

Kubernetes 1.9.0 or above with the admissionregistration.k8s.io/v1beta1 API enabled. Verify that by the following command:

kubectl api-versions | grep admissionregistration.k8s.io/v1beta1

The result should be:

admissionregistration.k8s.io/v1beta1

In addition, the MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers should be added and listed in the correct order in the admission-control flag of kube-apiserver.

Build

  1. Setup dep

    The repo uses dep as the dependency management tool for its Go codebase. Install dep by the following command:

go get -u github.com/golang/dep/cmd/dep
./build

Deploy

  1. Create namespace
kubectl apply -f deployment/namespace.yaml
  1. Create a signed cert/key pair and store it in a Kubernetes secret that will be consumed by sidecar deployment
./deployment/webhook-create-signed-cert.sh \
    --service gpu-mutating-webhook-svc \
    --secret gpu-mutating-webhook-certs \
    --namespace gpu-mutating
  1. Patch the MutatingWebhookConfiguration by set caBundle with correct value from Kubernetes cluster
cat deployment/mutatingwebhook.yaml | \
    deployment/webhook-patch-ca-bundle.sh > \
    deployment/mutatingwebhook-ca-bundle.yaml
  1. Deploy resources
kubectl apply -f deployment/deployment.yaml
kubectl apply -f deployment/service.yaml
kubectl apply -f deployment/mutatingwebhook-ca-bundle.yaml

Verify

  1. The sidecar inject webhook should be running
# kubectl get pods -n gpu-mutating
NAME                                               READY     STATUS    RESTARTS   AGE
gpu-mutating-webhook-deployment-7f55d45d64-mpkmk   1/1       Running   0          6m
# kubectl get deployment -n gpu-mutating
NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
gpu-mutating-webhook-deployment   1         1         1            1           7m
  1. Deploy an app in Kubernetes cluster, take sleep app as an example
# cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: sleep
    spec:
      containers:
      - name: sleep
        image: tutum/curl
        command: ["/bin/sleep","infinity"]
        imagePullPolicy: 
EOF
  1. Verify container env injected
# kubectl get pods
NAME                                                READY     STATUS      RESTARTS   AGE
sleep-854884fd47-98xq7                              1/1       Running     0          7m
# kubectl describe pod sleep-854884fd47-98xq7 | grep NVIDIA_VISIBLE_DEVICES
     NVIDIA_VISIBLE_DEVICES:  none

LICENSE

This is created from kube-mutating-webhook-tutorial

About

Kubernetes GPU Mutating Webhook

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Go 69.5%
  • Shell 29.7%
  • Dockerfile 0.8%