Skip to content

Commit

Permalink
fix(logs): make built-in metadata consistent between fluentd and otel
Browse files Browse the repository at this point in the history
  • Loading branch information
Mikołaj Świątek committed Jan 19, 2023
1 parent d24928f commit 2760953
Show file tree
Hide file tree
Showing 12 changed files with 112 additions and 49 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
to be consistent with Fluentd and chart v2.
- fix(logs): correctly handle newlines [#2805]
- Fixes [#2802], [#2803]
- fix(logs): make built-in metadata consistent between fluentd and otel [#2801]

[#2724]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2724
[#2745]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2745
Expand All @@ -55,6 +56,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
[#2802]: https://github.com/SumoLogic/sumologic-kubernetes-collection/issues/2802
[#2803]: https://github.com/SumoLogic/sumologic-kubernetes-collection/issues/2803
[#2805]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2805
[#2801]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2801
[v1.15.3-sumo-0]: https://github.com/SumoLogic/sumologic-kubernetes-fluentd/releases/tag/v1.15.3-sumo-0
[Unreleased]: https://github.com/SumoLogic/sumologic-kubernetes-collection/compare/v3.0.0-beta.0...main

Expand Down
6 changes: 3 additions & 3 deletions deploy/helm/sumologic/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `sumologic.logs.multiline.first_line_regex` | Regular expression to match first line of multiline logs. | `^\[?\d{4}-\d{1,2}-\d{1,2}.\d{2}:\d{2}:\d{2}` |
| `sumologic.logs.systemd.enabled` | Enable collecting systemd logs from Kubernets nodes. | `true` |
| `sumologic.logs.systemd.units` | List of systemd units to collect logs from. | See [values.yaml] |
| `sumologic.logs.container.sourceHost` | Set the _sourceHost metadata field in Sumo Logic. | `"%{k8s.pod.hostname}"` |
| `sumologic.logs.container.sourceName` | Set the _sourceName metadata field in Sumo Logic. | `"%{k8s.namespace.name}.%{k8s.pod.name}.%{k8s.container.name}"` |
| `sumologic.logs.container.sourceCategory` | Set the _sourceCategory metadata field in Sumo Logic. | `"%{k8s.namespace.name}/%{k8s.pod.pod_name}"` |
| `sumologic.logs.container.sourceHost` | Set the _sourceHost metadata field in Sumo Logic. | `""` |
| `sumologic.logs.container.sourceName` | Set the _sourceName metadata field in Sumo Logic. | `"%{namespace}.%{pod}.%{container}"` |
| `sumologic.logs.container.sourceCategory` | Set the _sourceCategory metadata field in Sumo Logic. | `"%{namespace}/%{pod_name}"` |
| `sumologic.logs.container.sourceCategoryPrefix` | Set the prefix, for _sourceCategory metadata. | `"kubernetes/"` |
| `sumologic.logs.container.sourceCategoryReplaceDash` | Used to replace - with another character. | `"/"` |
| `sumologic.logs.container.excludeContainerRegex` | A regular expression for container names. Logs from matching containers will not be sent to Sumo. | `""` |
Expand Down
35 changes: 25 additions & 10 deletions deploy/helm/sumologic/conf/logs/otelcol/config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -147,10 +147,23 @@ processors:
attributes:
- pattern: ^pod_annotations_.*
action: delete
resource/remove_k8s_pod_pod_name:
resource/remove_pod_name:
attributes:
- action: delete
key: k8s.pod.pod_name
key: pod_name
# As sourceprocessor can't set these attributes to be empty, we do it here instead
# If they're defined to be anything else in sourceprocessor, those values will overwrite these
resource/set_empty_source_metadata:
attributes:
- key: _sourceCategory
action: insert
value: ""
- key: _sourceHost
action: insert
value: ""
- key: _sourceName
action: insert
value: ""
groupbyattrs/containers:
keys:
- k8s.container.id
Expand Down Expand Up @@ -200,8 +213,8 @@ processors:
k8s.node.name: {{ .Values.sumologic.logs.container.excludeHostRegex | quote }}
annotation_prefix: "pod_annotations_"
pod_template_hash_key: "pod_labels_pod-template-hash"
pod_name_key: "k8s.pod.pod_name"
pod_key: "k8s.pod.name"
pod_name_key: "pod_name"
pod_key: "pod"
container_annotations:
enabled: {{ .Values.sumologic.logs.container.perContainerAnnotationsEnabled }}
prefixes: {{ toJson .Values.sumologic.logs.container.perContainerAnnotationPrefixes }}
Expand Down Expand Up @@ -366,17 +379,18 @@ service:
- groupbyattrs/containers
- k8s_tagger
- resource/add_cluster
- resource/set_empty_source_metadata
- resource/containers_copy_node_to_host
- sumologic_schema
- source/containers
{{- if .Values.sumologic.logs.container.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.container.otelcol.extraProcessors }}
{{ printf "- %s" ( $processor | keys | first ) | indent 8 }}
{{- end }}
{{- end }}
- resource/remove_k8s_pod_pod_name
- resource/remove_pod_name
- resource/drop_annotations
- attributes/remove_fluent_tag
- resource/containers_copy_node_to_host
- sumologic_schema
- batch
exporters:
- sumologic/containers
Expand Down Expand Up @@ -438,17 +452,18 @@ service:
- groupbyattrs/containers
- k8s_tagger
- resource/add_cluster
- resource/set_empty_source_metadata
- resource/containers_copy_node_to_host
- sumologic_schema
- source/containers
- logstransform/containers_parse_json
{{- if .Values.sumologic.logs.container.otelcol.extraProcessors }}
{{- range $processor := .Values.sumologic.logs.container.otelcol.extraProcessors }}
{{ printf "- %s" ( $processor | keys | first ) | indent 8 }}
{{- end }}
{{- end }}
- resource/remove_k8s_pod_pod_name
- resource/remove_pod_name
- resource/drop_annotations
- resource/containers_copy_node_to_host
- sumologic_schema
- batch
exporters:
- sumologic/containers
Expand Down
6 changes: 3 additions & 3 deletions deploy/helm/sumologic/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -258,11 +258,11 @@ sumologic:
extraProcessors: []

## Set the _sourceHost metadata field in Sumo Logic.
sourceHost: "%{k8s.pod.hostname}"
sourceHost: ""
## Set the _sourceName metadata field in Sumo Logic.
sourceName: "%{k8s.namespace.name}.%{k8s.pod.name}.%{k8s.container.name}"
sourceName: "%{namespace}.%{pod}.%{container}"
## Set the _sourceCategory metadata field in Sumo Logic.
sourceCategory: "%{k8s.namespace.name}/%{k8s.pod.pod_name}"
sourceCategory: "%{namespace}/%{pod_name}"
## Set the prefix, for _sourceCategory metadata.
sourceCategoryPrefix: "kubernetes/"
## Used to replace - with another character.
Expand Down
22 changes: 16 additions & 6 deletions docs/collecting-container-logs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,22 +68,33 @@ sumologic:
logs:
container:
## Set the _sourceHost metadata field in Sumo Logic.
sourceHost: "%{k8s.pod.hostname}"
sourceHost: ""
## Set the _sourceName metadata field in Sumo Logic.
sourceName: "%{k8s.namespace.name}.%{k8s.pod.name}.%{k8s.container.name}"
sourceName: "%{namespace}.%{pod}.%{container}"
## Set the _sourceCategory metadata field in Sumo Logic.
sourceCategory: "%{k8s.namespace.name}/%{k8s.pod.pod_name}"
sourceCategory: "%{namespace}/%{pod_name}"
## Set the prefix, for _sourceCategory metadata.
sourceCategoryPrefix: "kubernetes/"
## Used to replace - with another character.
sourceCategoryReplaceDash: "/"
```

As can be seen in the above example, these fields can contain templates of the form `%{field_name}`, where `field_name` is the name
of a resource attribute. Available resource attributes include [OpenTelemetry Kubernetes resource attributes][opentelemetry_k8s],
in addition to the following:
of a resource attribute. Available resource attributes include the values of `sumologic.logs.fields`, which by default are:

- `cluster`
- `container`
- `daemonset`
- `deployment`
- `host`
- `namespace`
- `node`
- `pod`
- `service`
- `statefulset`

in addition to the following:

- `_collector`
- `pod_labels_*` where `*` is the Pod label name

Expand Down Expand Up @@ -301,7 +312,6 @@ sumologic:
[configuration]: https://github.com/SumoLogic/sumologic-otel-collector/blob/main/docs/configuration.md
[values]: /deploy/helm/sumologic/values.yaml
[source_name]: https://help.sumologic.com/docs/send-data/reference-information/metadata-naming-conventions/#Source_Name
[opentelemetry_k8s]: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/semantic_conventions/k8s.md
[filter_processor_docs]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.69.0/processor/filterprocessor/README.md
[opentelemetry_processors]: https://opentelemetry.io/docs/collector/configuration/#processors
[attributes_processor_docs]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.69.0/processor/attributesprocessor/README.md
Expand Down
32 changes: 22 additions & 10 deletions tests/helm/metadata_logs_otc/static/otel.output.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -246,10 +246,21 @@ data:
attributes:
- action: delete
pattern: ^pod_annotations_.*
resource/remove_k8s_pod_pod_name:
resource/remove_pod_name:
attributes:
- action: delete
key: k8s.pod.pod_name
key: pod_name
resource/set_empty_source_metadata:
attributes:
- action: insert
key: _sourceCategory
value: ""
- action: insert
key: _sourceHost
value: ""
- action: insert
key: _sourceName
value: ""
source/containers:
annotation_prefix: pod_annotations_
collector: kubernetes
Expand All @@ -261,14 +272,14 @@ data:
k8s.namespace.name: ""
k8s.node.name: ""
k8s.pod.name: ""
pod_key: k8s.pod.name
pod_name_key: k8s.pod.pod_name
pod_key: pod
pod_name_key: pod_name
pod_template_hash_key: pod_labels_pod-template-hash
source_category: '%{k8s.namespace.name}/%{k8s.pod.pod_name}'
source_category: '%{namespace}/%{pod_name}'
source_category_prefix: kubernetes/
source_category_replace_dash: /
source_host: '%{k8s.pod.hostname}'
source_name: '%{k8s.namespace.name}.%{k8s.pod.name}.%{k8s.container.name}'
source_host: ""
source_name: '%{namespace}.%{pod}.%{container}'
source/kubelet:
collector: kubernetes
source_category: kubelet
Expand Down Expand Up @@ -310,12 +321,13 @@ data:
- groupbyattrs/containers
- k8s_tagger
- resource/add_cluster
- resource/set_empty_source_metadata
- resource/containers_copy_node_to_host
- sumologic_schema
- source/containers
- logstransform/containers_parse_json
- resource/remove_k8s_pod_pod_name
- resource/remove_pod_name
- resource/drop_annotations
- resource/containers_copy_node_to_host
- sumologic_schema
- batch
receivers:
- otlp
Expand Down
26 changes: 19 additions & 7 deletions tests/helm/metadata_logs_otc/static/templates.output.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -246,10 +246,21 @@ data:
attributes:
- action: delete
pattern: ^pod_annotations_.*
resource/remove_k8s_pod_pod_name:
resource/remove_pod_name:
attributes:
- action: delete
key: k8s.pod.pod_name
key: pod_name
resource/set_empty_source_metadata:
attributes:
- action: insert
key: _sourceCategory
value: ""
- action: insert
key: _sourceHost
value: ""
- action: insert
key: _sourceName
value: ""
source/containers:
annotation_prefix: pod_annotations_
collector: my_collectorName
Expand All @@ -261,8 +272,8 @@ data:
k8s.namespace.name: my_containers_excludeNamespaceRegex
k8s.node.name: my_containers_excludeHostRegex
k8s.pod.name: my_containers_excludePodRegex
pod_key: k8s.pod.name
pod_name_key: k8s.pod.pod_name
pod_key: pod
pod_name_key: pod_name
pod_template_hash_key: pod_labels_pod-template-hash
source_category: my_containers_sourceCategory
source_category_prefix: my_containers_sourceCategoryPrefix
Expand Down Expand Up @@ -310,12 +321,13 @@ data:
- groupbyattrs/containers
- k8s_tagger
- resource/add_cluster
- resource/set_empty_source_metadata
- resource/containers_copy_node_to_host
- sumologic_schema
- source/containers
- logstransform/containers_parse_json
- resource/remove_k8s_pod_pod_name
- resource/remove_pod_name
- resource/drop_annotations
- resource/containers_copy_node_to_host
- sumologic_schema
- batch
receivers:
- otlp
Expand Down
19 changes: 15 additions & 4 deletions tests/integration/helm_fluentbit_fluentd_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"context"
"fmt"
"sort"
"strings"
"testing"
"time"

Expand Down Expand Up @@ -310,10 +311,20 @@ func Test_Helm_FluentBit_Fluentd(t *testing.T) {
"host": internal.NodeNameRegex,
"master_url": ".+",
"node": internal.NodeNameRegex,
// The values below are a result of a bug and should be fixed shortly, see https://github.com/SumoLogic/sumologic-kubernetes-collection/issues/2767
"_sourceName": "undefined.undefined.undefined",
"_sourceCategory": "kubernetes/undefined/undefined",
"_sourceHost": "undefined",
"_sourceName": fmt.Sprintf(
"%s\\.%s%s\\.%s",
internal.LogsGeneratorNamespace,
internal.LogsGeneratorName,
internal.PodDeploymentSuffixRegex,
internal.LogsGeneratorName,
),
"_sourceCategory": fmt.Sprintf(
"%s/%s/%s", // dashes instead of hyphens due to sourceCategoryReplaceDash
internal.ClusterName,
strings.ReplaceAll(internal.LogsGeneratorNamespace, "-", "/"),
strings.ReplaceAll(internal.LogsGeneratorName, "-", "/"), // this is the pod name prefix, in this case the Deployment name
),
"_sourceHost": internal.EmptyRegex,
},
internal.ReceiverMockNamespace,
internal.ReceiverMockServiceName,
Expand Down
Loading

0 comments on commit 2760953

Please sign in to comment.