Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(otellogs)!: add config.merge option #2652

Merged
merged 1 commit into from
Nov 28, 2022
Merged

feat(otellogs)!: add config.merge option #2652

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- feat(logs)!: simplify metadata configuration [#2626]
- feat(metrics)!: simplify metadata configuration [#2622]
- feat(events)!: add config.merge option [#2643]
- feat(otellogs)!: add config.merge option [#2652]

### Changed

Expand Down Expand Up @@ -88,6 +89,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
[#2622]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2622
[#2643]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2643
[#2654]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2654
[#2652]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2652
[Unreleased]: https://github.com/SumoLogic/sumologic-kubernetes-collection/compare/v2.17.0...main
[telegraf_operator_comapare_1.3.5_and_1.3.10]: https://github.com/influxdata/helm-charts/compare/telegraf-operator-1.3.5...telegraf-operator-1.3.10
[cert-manager-1.4]: https://github.com/cert-manager/cert-manager/releases/tag/v1.4.0
Expand Down
9 changes: 7 additions & 2 deletions deploy/helm/sumologic/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -331,8 +331,13 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `otelcol.config.processors.batch.send_batch_max_size` | Sets the maximum allowed size of a batch (in number of spans). Use with caution, setting too large value might cause 413 Payload Too Large errors. | `512` |
| `otelcol.logLevelFilter` | Do not send otelcol logs if `true`. | `false` |
| `otelgateway.deployment.nodeSelector` | Node selector for otelgateway deployment. [See docs/best-practices.md for more information.](/docs/best-practices.md) | `{}` |
| `otellogs.daemonset.nodeSelector` | Node selector for otellogs daemonset. [See docs/best-practices.md for more information.](/docs/best-practices.md) | `{}` |
| `otellogs.daemonset.priorityClassName` | Priority class name for OpenTelemetry Agent log pods. | If not provided then set to `RELEASE-NAME-sumologic-priorityclass`. |
| `otellogs.image.repository` | Image repository for otelcol docker container. | `public.ecr.aws/sumologic/sumologic-otel-collector` |
| `otellogs.image.tag` | Image tag for otelcol docker container. | `0.54.0-sumo-0` |
| `otellogs.image.pullPolicy` | Image pullPolicy for otelcol docker container. | `IfNotPresent` |
| `otellogs.logLevel` | Log level for the OpenTelemtry Collector. Can be `debug`, `info`, `warn`, `error`, `dpanic`, `panic`, `fatal`. | `info` | | `{}` |
| `otellogs.config.merge` | Configuration for log collector otelcol, merged with defaults. See also https://github.com/SumoLogic/sumologic-otel-collector/blob/main/docs/configuration.md. | {} |
| `otellogs.config.override` | Configuration for log collector otelcol, replaces defaults. See also https://github.com/SumoLogic/sumologic-otel-collector/blob/main/docs/configuration.md. | {} |
| `otellogs.daemonset` | OpenTelemetry Collector Daemonset customization options. See values.yaml for more details. | See [values.yaml] |
| `metadata.image.repository` | Image repository for otelcol docker container. | `public.ecr.aws/sumologic/sumologic-otel-collector` |
| `metadata.image.tag` | Image tag for otelcol docker container. | `0.0.18` |
| `metadata.image.pullPolicy` | Image pullPolicy for otelcol docker container. | `IfNotPresent` |
Expand Down
14 changes: 10 additions & 4 deletions deploy/helm/sumologic/templates/logs/collector/otelcol/configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,13 @@
{{- if eq (include "logs.collector.otelcol.enabled" .) "true" }}
{{ $baseConfig := (tpl (.Files.Get "conf/logs/collector/otelcol/config.yaml") .) | fromYaml }}
{{ $mergeConfig := .Values.otellogs.config.merge }}
{{ $overrideConfig := .Values.otellogs.config.override }}
{{ $finalConfig := "" }}
{{ if $overrideConfig }}
{{ $finalConfig = $overrideConfig }}
{{ else }}
{{ $finalConfig = mergeOverwrite $baseConfig $mergeConfig }}
{{ end }}
apiVersion: v1
kind: ConfigMap
metadata:
Expand All @@ -7,9 +16,6 @@ metadata:
app: {{ template "sumologic.labels.app.logs.collector.configmap" . }}
{{- include "sumologic.labels.common" . | nindent 4 }}
data:
{{- $baseConfig := (tpl (.Files.Get "conf/logs/collector/otelcol/config.yaml") .) | fromYaml -}}
{{- $overrideConfig := .Values.otellogs.config.override -}}
{{- $finalConfig := mergeOverwrite $baseConfig $overrideConfig }}
config.yaml: |
{{- $finalConfig | toYaml | nindent 4 }}
{{- $finalConfig | toYaml | nindent 4 }}
{{- end }}
21 changes: 16 additions & 5 deletions deploy/helm/sumologic/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4151,11 +4151,6 @@ otelevents:
# readOnly: true

## Configure log collection with Otelcol
## This is currently highly experimental and not recommended unless you really know what you're doing.
## In particular, this configuration currently only covers container logs, and only supports docker-shim.
## Unlike the fluent-bit configuration, it also reads logs from /var/log/pods instead of /var/log/containers
## which is consistent with CRI, but may possibly cause issues on older K8s versions.
## This is an alpha feature, and may change in the near future.
otellogs:

## Metrics from Collector
Expand All @@ -4174,6 +4169,22 @@ otellogs:
logLevel: info

config:
## Directly alter the OT configuration. The value of this key should be a dictionary, that will
## be directly merged with the generated configuration, overriding existing values.
## For example:
# override:
# processors:
# batch:
# send_batch_size: 512
## will change the batch size of the pipeline.
##
## WARNING: This field is not subject to backwards-compatibility guarantees offered by the rest
## of this chart. It involves implementation details that may change even in minor versions.
## Use with caution, and consider opening an issue, so your customization can be added in a safer way.
merge: {}
## Completely override existing config and replace it with the contents of this value.
## The value of this key should be a dictionary, that will replace the normal configuration.
## This is an advanced feature, use with caution, and review the generated configuration first.
override: {}

daemonset:
Expand Down
2 changes: 1 addition & 1 deletion docs/opentelemetry-collector.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ versions while preserving the same end-to-end behaviour. Use of this field is di
the necessary customizations should be able to be achieved without touching the otel configuration
directly. Please open an issue if your use case requires the use of this field.

The `metadata.metrics.config.override` and `otellogs.config.override` keys can be used to provide configuration that will be completely
The `metadata.logs.config.override` and `otellogs.config.override` keys can be used to provide configuration that will be completely
replace the default configuration. As above, care must be taken not to depend on implementation details
that may change between minor releases of this Chart.

Expand Down
2 changes: 1 addition & 1 deletion .../helm/logs_otc/static/override.input.yaml → tests/helm/logs_otc/static/merge.input.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ fluent-bit:

otellogs:
config:
override:
merge:
receivers:
journald:
units:
Expand Down
0 ...helm/logs_otc/static/override.output.yaml → tests/helm/logs_otc/static/merge.output.yaml
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion tests/integration/values/values_helm_default_otelcol_logs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ fluent-bit:

otellogs:
config:
override:
merge:
receivers:
journald:
directory: /run/log/journal
Expand Down
85 changes: 85 additions & 0 deletions vagrant/opentelemetry-collector.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: prometheus
namespace: sumologic
spec:
mode: statefulset
targetAllocator:
enabled: true
allocationStrategy: consistent-hashing
prometheusCR:
enabled: true
image: ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:0.64.1
serviceAccount: prometheus-collector
replicas: 3
config: |
receivers:
prometheus:
config:
target_allocator:
endpoint: http://prometheus-targetallocator
interval: 30s
collector_id: ${POD_NAME}
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s

exporters:
logging:

service:
pipelines:
metrics:
receivers: [prometheus]
processors: []
exporters: [logging]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus-collector
namespace: sumologic
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: prometheus-collector
namespace: sumologic
rules:
- apiGroups:
- "*"
resources:
- "*"
verbs:
- "*"
- apiGroups:
- "monitoring.coreos.com"
resources:
- "*"
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus-collector
namespace: sumologic
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: prometheus-collector
namespace: sumologic