[gloo-fed] Support kubeconfig with exec command

[shahar-h]

Version

See label (1.14)

Is your feature request related to a problem? Please describe.

Background: We want gloo-fed to use kubeconfig with exec command to avoid using a static token in our cluster.
For this to work we need to mount some binary into gloo-fed container.

In order to mount this binary we need gloo-fed chart to support the following configurations in chart values:

• Init containers for gloo-fed deployment
• Volumes for gloo-fed deployment
• VolumeMounts for gloo-fed container

Currently we maintain our own fork of gloo-fed chart in order to achieve that, and we'll be happy to get rid of it.

Describe the solution you'd like

Support the configurations described above.

Describe alternatives you've considered

No response

Additional Context

No response

[sheidkamp]

The changes in the fork look like:

diff --git a/install/helm/gloo-fed/templates/gloo-fed-deployment.yaml b/install/helm/gloo-fed/templates/gloo-fed-deployment.yaml
index b2de7b0df..6b74b1791 100644
--- a/install/helm/gloo-fed/templates/gloo-fed-deployment.yaml
+++ b/install/helm/gloo-fed/templates/gloo-fed-deployment.yaml
@@ -99,6 +99,10 @@ spec:
           - name: http-monitoring
             containerPort: 9091
         {{- end }}
+        {{- if $glooFed.sharedVolumes }}
+        volumeMounts:
+{{ toYaml $glooFed.sharedVolumes.volumeMounts | indent 10 }}
+        {{- end }}
 {{- if $glooFed.resources }}
         resources:
 {{ toYaml $glooFed.resources | indent 10}}
@@ -109,6 +113,7 @@ spec:
             memory: 256Mi
         {{- end}}
         securityContext:
+          runAsNonRoot: true
           readOnlyRootFilesystem: true
           allowPrivilegeEscalation: false
           capabilities:
@@ -118,6 +123,17 @@ spec:
       imagePullSecrets:
       - name: {{ $glooFedImage.pullSecret }}
   {{- end}}
+      securityContext:
+        runAsNonRoot: true
+      {{- if $glooFed.initContainers }}
+      initContainers:
+{{ toYaml $glooFed.initContainers | indent 6 }}
+      {{- end}}
+      {{- if $glooFed.sharedVolumes }}
+      volumes:
+{{ toYaml $glooFed.sharedVolumes.volumes | indent 6 }}
+      {{- end }}
+
 
 
 ---
diff --git a/install/helm/gloo-fed/templates/gloo-fed-rbac.yaml b/install/helm/gloo-fed/templates/gloo-fed-rbac.yaml
index 274635a0c..0d7e8741d 100644
--- a/install/helm/gloo-fed/templates/gloo-fed-rbac.yaml
+++ b/install/helm/gloo-fed/templates/gloo-fed-rbac.yaml
@@ -120,5 +120,38 @@ roleRef:
   kind: ClusterRole
   name: gloo-fed-{{ .Release.Namespace }}
   apiGroup: rbac.authorization.k8s.io
+
+
+{{- if .Values.glooFed.roleRules }}
+---
+
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: gloo-fed
+  namespace: {{ .Release.Namespace }}
+  labels:
+    app: gloo-fed
+rules:
+{{ toYaml .Values.glooFed.roleRules }}
+
+---
+
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: gloo-fed
+  namespace: {{ .Release.Namespace }}
+  labels:
+    app: gloo-fed
+subjects:
+- kind: ServiceAccount
+  name: gloo-fed
+roleRef:
+  kind: Role
+  name: gloo-fed
+  apiGroup: rbac.authorization.k8s.io
+{{- end }}
+##############################
 {{- end }}
 {{- end }} {{/* .Values.enabled */}}

[sam-heilbron]

I think we should try to standardize our deployment templates as part of this effort. We currently support InitContainers on the PodSpec. I would recommend starting to see where that is consumed, and if its possible to introduce that to the other deployments. After that, we could explore a "ContainerSpec", where we expose the options available to containers.

If this isn't possible given the current helm charts, perhaps we write up a feature request to standardize our templates

[sheidkamp]

Path forward:

• Embed glooGen.DeploymentSpec in gloo-fed deployment containers
• Extend current securityContext customization approach to include gloo-fed containers
  • Add a merge field to the SecurityContext structures so we can easily just add the runAsNonRoot field
  • Requires changes in gloo and gloo-ee
  • Note - square up with the gloo deployments should be configured by default to run as "runAsNonRoot: true" in all level (spec and container level) #6423
• Add in Volume/VolumeMount fields basically as-is
• Add in RBAC roles basically as-is

(and of course all expected tests an documentation)

[sheidkamp]

Here's a sample values.yaml file that should replicate the behavior of the fork on the new releases:

gloo-fed:
  glooFed:
    initContainers:
    - name: busybox1
      image: busybox
      command: ["/bin/sh"]
      args: ["-c", "echo '<html><h1>Hi I am from Init container</h1>' >> /work-dir/index.html"]
    glooFed:
      securityContext:
        runAsNonRoot: true
        mergePolicy: helm-merge
      volumeMounts:
      - mountPath: /etc/glooFed-mount
        name: shared-mount
    volumes:
    - name: shared-mount
    roleRules:
      - apiGroups: ["*"]
        resources: ["*"]
        verbs: ["*"]

Generates the following YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: gloo-fed
  name: gloo-fed
  namespace: default
spec:
  selector:
    matchLabels:
      app: gloo-fed
  replicas: 1
  template:
    metadata:
      labels:
        app: gloo-fed
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "9091"
        prometheus.io/scrape: "true"
    spec:
      serviceAccountName: gloo-fed
      initContainers: 
        - args:
          - -c
          - echo '<html><h1>Hi I am from Init container</h1>' >> /work-dir/index.html
          command:
          - /bin/sh
          image: busybox
          name: busybox1
      containers:
      - image: quay.io/solo-io/gloo-fed:1.0.0-sah
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: WRITE_NAMESPACE
          value: default
        - name: GLOO_LICENSE_KEY
          valueFrom:
            secretKeyRef:
              name: license
              key: license-key
        - name: START_STATS_SERVER
          value: "true"
        - name: CW_REMOTE_RETRY_TYPE
          value: "backoff"
        - name: CW_REMOTE_RETRY_DELAY
          value: "1s"
        - name: CW_REMOTE_RETRY_MAX_DELAY
          value: "0"
        - name: CW_REMOTE_RETRY_MAX_JITTER
          value: "100ms"
        - name: CW_REMOTE_RETRY_ATTEMPTS
          value: "0"
        - name: CW_LOCAL_RETRY_TYPE
          value: "backoff"
        - name: CW_LOCAL_RETRY_DELAY
          value: "100ms"
        - name: CW_LOCAL_RETRY_MAX_DELAY
          value: "0"
        - name: CW_LOCAL_RETRY_MAX_JITTER
          value: "100ms"
        - name: CW_LOCAL_RETRY_ATTEMPTS
          value: "5"
        imagePullPolicy: IfNotPresent
        name: gloo-fed
        volumeMounts:
          - mountPath: /etc/glooFed-mount
            name: shared-mount
        resources:
          requests:
            cpu: 125m
            memory: 256Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
      volumes:
      - name: shared-mount


---
# Source: gloo-ee/charts/gloo-fed/templates/gloo-fed-rbac.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gloo-fed
  namespace: default
  labels:
    app: gloo-fed
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
---
# Source: gloo-ee/charts/gloo-fed/templates/gloo-fed-rbac.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gloo-fed
  namespace: default
  labels:
    app: gloo-fed
subjects:
- kind: ServiceAccount
  name: gloo-fed
roleRef:
  kind: Role
  name: gloo-fed
  apiGroup: rbac.authorization.k8s.io
---

[sheidkamp]

will be part of gloo-fed/solo-projects v1.14.2 and v1.15.0-beta2

[sheidkamp]

Reopened because we missed the podSecurityContext.