Otel statefulset cloudwatch collector

[rnishtala-sumo]

This PR creates a statefulset for collecting logs from cloudwatch. It also documents some of the steps needed to setup log ingestion from cloudwatch. The fargate readme has been updated with some details

• It has initial documentation on how to set this up

  • https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/otel-statefulset-collector/docs/fargate.md#logs

• The changes have been tested on a fargate cluster, with logs being ingested into sumo

  • example-1: https://stagdata.long.sumologic.net/ui/#/search/create?id=BKYd1rzPGnOcpDlsB3D1twcoSk6xBYIxl36XrUF8
  • example-2: https://stagdata.long.sumologic.net/ui/#/search/create?id=MIOPkMUGxmQOVyI4tZntY8q6Bgzq8s73HR8wNdKc

• there isn't any processing or metadata enrichment yet

• There's a basic template test, integration tests are pending

https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/otel-statefulset-collector/docs/fargate.md#logs

Checklist

• Changelog updated or skip changelog label added
• Documentation updated
• Template tests added for new features
• Integration tests added or modified for major features

[aboguszewski-sumo]

Some additional thoughts after using this:

• we should probably decide which log groups we're going to observe: with our suggested fluent-bit config there's going to be only one group, can it be more groups with different configs?
• current way of providing awscloudwatch receiver config is a bit tiresome, for example I tried to use the following config to read log from a specific group:

sumologic:
  logs:
    collector:
      otelcloudwatch:
        awscloudwatch:
          logs:
            groups:
              named:
                fluent-bit-cloudwatch:
                  names: ["some-name"]

and it raised an error, because the default is also using autodiscover, and both autodiscover and named can't be used at the same time. So we should probably either have this receiver's config empty as default or don't copy it directly as it was suggested in one of the comments.

• persistency for the collector definitely should be configurable (disabling it manually is a mess) and it should be explained in more detail how to enable it on Fargate
• in general, we should have some common section about how to enable persistency on Fargate that will be common for the collector, metrics and logs

[sumo-drosiek]

Overally LGTM, some minor minor comments

[rnishtala-sumo]

I've made the suggested changes, will keep this documentation task below open:

https://jira.kumoroku.com/jira/browse/SUMO-205563

Added the comment from this PR to the above ticket: #2982 (comment)

[sumo-drosiek]

Please doucment it in README as github actions are failing, also, I think it's not otellogs anymore, eventually otellogs.cloudwatch or otelcloudwatch

[sumo-drosiek]

LGTM