Expose http sources for metrics in values.yaml

[sumo-drosiek]

Description

Expose http sources in values.yaml

ToDO

• Move out configuration from configmap to separate files (like fluentd conf)
• Discuss if logs/events/default metrics should be exposed
  I decided to not expose logs and events so far
• Add enable/disable flag
• Dynamic generation of fluentd configuration(?)

Testing performed

• ci/build.sh
• Redeploy fluentd and fluentd-events pods
• Confirm events, logs, and metrics are coming in

[perk-sumo]

Overall I like this but as discussed some UX changes are needed.

[sumo-drosiek]

I updated the PR with more user-friendly configuration, but this is highly breaking change for non-helm users

[sumo-drosiek]

Here is example of adding new source to sumologic and fluentd:

    --set sumologic.sources.new_source.value="my new source" \ # create new source named my new source and available in helm as new_source
    --set fluentd.metrics.output.new_source.match="test_match" \ # fluentd <match test_match> clausule
    --set fluentd.metrics.output.new_source.id="test_id" \ # fluentd @id parameter
    --set fluentd.metrics.output.new_source.weight=0 \ # optional, default is 0, order of fluentd match (lower, earlier)
    --set fluentd.metrics.output.new_source.drop=true # optional, default is false, if true, drop traffic instead of passing it to the sumologic \
    --set fluentd.buffer.filePaths.metrics.new_source=/fluentd/buffer/metrics.apiserver # set persistence

[sumo-drosiek]

Unfortunately if source value is changed, the new source is created via terraform and old one is not cleaned up

I suppose persistence setup job could fix it

[sumo-drosiek]

Fixes prometheus issue. It has been saying that configuration for http://$(CHART).$(NAMESPACE).svc.cluster.local:9888/prometheus.metrics.control-plane is duplicated

[frankreno]

The only thing i find misleading with this setup is that we collect from other control plane components and do not prefix them with control-plane. Any reason to just remove the control-plane part?

[sumo-drosiek]

Yes, the fluentd configuration will be more complex: either more complex regex for tags or output section per endpoint

[frankreno]

How many metrics is each of these producing? We need to be conscious of blacklisting and if we are close to the limit, we should distribute across multiiple HTTP sources to prevent that.