Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(helm): add helper function to add quotes to annotation_match #1655

Merged
merged 1 commit into from
Jun 14, 2021

Conversation

kasia-kujawa
Copy link
Contributor

@kasia-kujawa kasia-kujawa commented Jun 14, 2021

quote function is not use as it automatically adds \ before special characters

Example tested configuration:

values.yaml

fluentd:
  logs:
    containers:
      perContainerAnnotationsEnabled: true
      perContainerAnnotationPrefixes:
        - "sumologic.com/"
        - "tailing-sidecar.sumologic.com/"
        
  metadata:
   annotation_match:
    - 'sumologic\.com.*'
    - 'tailing-sidecar.sumologic\.com.*'
    - "test-sumologic\\.com.*"
    - "test-tailing-sidecar.sumologic\\.com.*"

pod1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod1
  namespace: sumologic
  annotations:
    sumologic.com/count1.sourceCategory: test-source-category51
    sumologic.com/count2.sourceCategory: test-source-category61
  labels:
    app: pod-with-annotations
spec:
  containers:
  - name: count1
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "example0: $i $(date)"
        echo "example1: $i $(date)"
        echo "example2: $i $(date)"
        i=$((i+1));
        sleep 5;
      done
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlogconfig
      mountPath: /varconfig/log
  - name: count2
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "example0: $i $(date)"
        echo "example1: $i $(date)"
        echo "example2: $i $(date)"
        i=$((i+1));
        sleep 5;
      done
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlogconfig
      mountPath: /varconfig/log
  volumes:
  - name: varlog
    emptyDir: {}
  - name: varlogconfig
    emptyDir: {}

pod2.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod2
  namespace: sumologic
  annotations:
    tailing-sidecar.sumologic.com/count1.sourceCategory: test-source-category31
    tailing-sidecar.sumologic.com/count2.sourceCategory: test-source-category41
  labels:
    app: pod-with-annotations
spec:
  containers:
  - name: count1
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "example0: $i $(date)"
        echo "example1: $i $(date)"
        echo "example2: $i $(date)"
        i=$((i+1));
        sleep 5;
      done
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlogconfig
      mountPath: /varconfig/log
  - name: count2
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "example0: $i $(date)"
        echo "example1: $i $(date)"
        echo "example2: $i $(date)"
        i=$((i+1));
        sleep 5;
      done
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlogconfig
      mountPath: /varconfig/log
  volumes:
  - name: varlog
    emptyDir: {}
  - name: varlogconfig
    emptyDir: {}

fluentd configuration in helm templates (helm --dry-run)

  logs.kubernetes.metadata.filter.conf: |-
    annotation_match '["sumologic\.com.*","tailing-sidecar.sumologic\.com.*","test-sumologic\.com.*","test-tailing-sidecar.sumologic\.com.*"]'
    de_dot false
    watch "true"
    ca_file ""
    verify_ssl "true"
    client_cert ""
    client_key ""
    bearer_token_file ""
    cache_size "10000"
    cache_ttl "7200"
    tag_to_kubernetes_name_regexp '.+?\.containers\.(?<pod_name>[^_]+)_(?<namespace>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$'

fluentd configuration captured by fluentd pod (kubectl logs collection-sumologic-fluentd-logs-2)

    <filter containers.**>
      @type kubernetes_metadata
      @log_level "error"
      annotation_match ["sumologic.com.*","tailing-sidecar.sumologic.com.*","test-sumologic.com.*","test-tailing-sidecar.sumologic.com.*"]
      de_dot false
      watch true
      ca_file ""
      verify_ssl true
      client_cert ""
      client_key ""
      bearer_token_file ""
      cache_size 10000
      cache_ttl 7200
      tag_to_kubernetes_name_regexp ".+?\\.containers\\.(?<pod_name>[^_]+)_(?<namespace>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\\.log$"
    </filter>

@kasia-kujawa kasia-kujawa force-pushed the kk-add-quotes-annotataion-match branch 3 times, most recently from cf3e5ad to a026924 Compare June 14, 2021 12:41
quote function is not use as it automatically adds '\' before special characters
@kasia-kujawa kasia-kujawa force-pushed the kk-add-quotes-annotataion-match branch from a026924 to f771448 Compare June 14, 2021 12:43
@kasia-kujawa kasia-kujawa marked this pull request as ready for review June 14, 2021 12:51
@kasia-kujawa kasia-kujawa requested a review from a team as a code owner June 14, 2021 12:51
{{- define "fluentd.metadata.annotations_match.quotes" -}}
{{- $matches_with_quotes := list -}}
{{- range $match := .Values.fluentd.metadata.annotation_match }}
{{- $match_with_quotes := printf "\"%s\"" $match }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use quote function as someone potentially can put quotes in variable

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested with quote and I noticed that it adds \ before special characters. I used this in following way:

{{- $match_with_quotes := $match  | quote }}

and then in help templates there was following form:

annotation_match ["sumologic\\.com.*","tailing-sidecar.sumologic\\.com.*","test-sumologic\\.com.*","test-tailing-sidecar.sumologic\\.com.*"]

and in fluend logs I saw that Pod had following configuration

 <filter containers.**>
      @type kubernetes_metadata
      @log_level "error"
      annotation_match ["sumologic\\.com.*","tailing-sidecar.sumologic\\.com.*","test-sumologic\\.com.*","test-tailing-sidecar.sumologic\\.com.*"]
      de_dot false
      watch true
      ca_file ""
      verify_ssl true
      client_cert ""
      client_key ""
      bearer_token_file ""
      cache_size 10000
      cache_ttl 7200
      tag_to_kubernetes_name_regexp ".+?\\.containers\\.(?<pod_name>[^_]+)_(?<namespace>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\\.log$"
    </filter>

Copy link
Contributor

@sumo-drosiek sumo-drosiek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@kasia-kujawa kasia-kujawa merged commit b433faa into main Jun 14, 2021
@kasia-kujawa kasia-kujawa deleted the kk-add-quotes-annotataion-match branch June 14, 2021 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants