feat(otellogs): read from end (#2710)

* feat(otellogs): read from end

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

* chore: changelog

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

* docs: add migration information

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

* Update docs/v3-migration-doc.md

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

CHANGELOG.md

@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - chore: upgrade Fluent Bit to v1.6.10-sumo-3 [#2712]
 - chore: upgrade otelcol to 0.66.0-sumo-0 [#2686] [#2687] [#2692] [#2693]
+- feat(otellogs): read from end [#2710]
 - fix(openshift): changed allowed fsgroups in SecurityContextConstraints [#2717]
 - fix(openshift): set securityContexts for otelcol-logs-collector [#2717]
 - fix: obey proxy settings in otelcol [#2719]
@@ -21,6 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 [#2687]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2687
 [#2693]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2693
 [#2692]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2692
+[#2710]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2710
 [#2712]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2712
 [#2717]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2717
 [#2719]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2719

deploy/helm/sumologic/conf/logs/collector/otelcol/config.yaml

@@ -42,7 +42,6 @@ receivers:
   filelog/containers:
     include:
       - /var/log/pods/*/*/*.log
-    start_at: beginning
     ## sets fingerprint_size to 17kb in order to match the longest possible docker line (which by default is 16kb)
     ## we want to include timestamp, which is at the end of the line
     fingerprint_size: 17408

docs/v3-migration-doc.md

@@ -9,8 +9,9 @@
     - [Otelcol StatefulSets](#otelcol-statefulsets)
     - [Tracing/Instrumentation changes](#tracinginstrumentation-changes)
     - [Additional Service Monitors](#additional-service-monitors)
-  - [Known issues](#known-issues)
-    - [Cannot delete pod if using Tailing Sidecar Operator](#cannot-delete-pod-if-using-tailing-sidecar-operator)
+- [Known issues](#known-issues)
+  - [Cannot delete pod if using Tailing Sidecar Operator](#cannot-delete-pod-if-using-tailing-sidecar-operator)
+  - [OpenTelemetry Collector doesn't read logs from the beginning of files](#opentelemetry-collector-doesnt-read-logs-from-the-beginning-of-files)
 
 Based on the feedback from our users, we will be introducing several changes
 to the Sumo Logic Kubernetes Collection solution.
@@ -89,6 +90,11 @@ In this document we detail the changes as well as the exact steps for migration.
 
 - Upgrading Tailing Sidecar Operator helm chart to v0.5.5. There is no breaking change if using annotations only.
 
+- OpenTelemetry Logs Collector will read from end of file now.
+
+  See [OpenTelemetry Collector doesn't read logs from the beginning of files](#opentelemetry-collector-doesnt-read-logs-from-the-beginning-of-files)
+  if you want to keep old behavior.
+
 - Changed `otelagent` from `DaemonSet` to `StatefulSet`
 
 - Moved parameters from `otelagent.*` to `otelcolInstrumentation.*`
@@ -269,3 +275,19 @@ Please try to remove pod later.
 [Falco documentation]: https://github.com/falcosecurity/charts/tree/falco-2.4.2/falco
 [metrics-server-upgrade]: https://github.com/bitnami/charts/tree/5b09f7a7c0d9232f5752840b6c4e5cdc56d7f796/bitnami/metrics-server#to-600
 [kube-prometheus-stack-image-migration]: https://github.com/prometheus-community/helm-charts/tree/kube-prometheus-stack-42.1.0/charts/kube-prometheus-stack#from-41x-to-42x
+
+#### OpenTelemetry Collector doesn't read logs from the beginning of files
+
+This is done by design. We are not going to read logs from time before the collection has been installed.
+
+In order to keep old behavior (can result in logs duplication for some cases), please use the following configuration:
+
+```
+metadata:
+  logs:
+    config:
+      merge:
+        receivers:
+          filelog/containers:
+            start_at: beginning
+```

tests/helm/logs_otc/static/basic.output.yaml

@@ -143,7 +143,6 @@ data:
         - from: body.log
           to: body
           type: move
-        start_at: beginning
         storage: file_storage
       journald:
         directory: /var/log/journal