Add filereload receiver

[newly12]

Description:

Add filereload receiver to support dynamically start/stop partial pipelines(receiver, processor) from configs.

Link to tracking Issue:
#15714

Testing:
UT.

Documentation:
Please check README doc.

[mx-psi]

This needs an issue and sponsor in accordance to the contributing guidelines for new components. Thanks!

[newly12]

Had filed issue. I can fix accordingly per the guidelines suggested if the idea looks good generally. Feel free to let me know if any question.

[tigrannajaryan]

This looks very interesting, however I think functionality like this may be more useful if broken down into 3 composable parts:

• Watching of configuration in external file(s). This should work for any part of the config, instead of being a feature of a particular receiver. We already have a mechanism for this, but it is not implemented by fileprovider config provider yet. We already allow including external config files, but we don't watch for changes.
• Pipelines that can be connected. We have an open issue for this Signal Translation Proprosal opentelemetry-collector#2336
• A receiver that can create other receivers. Need to figure out how it intersects with the existing receivercreator.

These bits can then be composed to result in the functionality that this PR suggests, but they can also be composed in other interesting ways.

@open-telemetry/collector-approvers @open-telemetry/collector-contrib-approvers thoughts?

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[newly12]

Thanks for the comment, I will take a look at confmap and see how it can be leveraged in this receiver and get back soon.

[newly12]

Watching of configuration in external file(s). This should work for any part of the config, instead of being a feature of a particular receiver. We already have a mechanism for this, but it is not implemented by fileprovider config provider yet. We open-telemetry/opentelemetry-collector#6276 including external config files, but we don't watch for changes.

open-telemetry/opentelemetry-collector#5945 for watch implementation in fileprovider, however it looks like the current mechanism only allows external config for a component(receiver, processor?, exporter), and on config changes entire service will be restarted(by reloadConfiguration), which is expensive if there is no change for some components, say exporters are not updated but only receiver configs are updated.

Pipelines that can be connected. We have an open issue for this open-telemetry/opentelemetry-collector#2336

open-telemetry/opentelemetry-collector#6372 for connector implementation.

A receiver that can create other receivers. Need to figure out how it intersects with the existing receivercreator.
I think either a receiver that creates other receivers or a mechanism starts/stops pipelines dynamically.

In fact the need is to spawn not only receivers but also processors, given usually different receivers(data source) require different processing logical, so maybe a mechanism to spawn receivers and processors(it will nice to be able to spawn exporter as well), it sounds like mechanism/ability to spawn pipelines dynamically will be needed.

[tigrannajaryan]

on config changes entire service will be restarted(by reloadConfiguration), which is expensive if there is no change for some components, say exporters are not updated but only receiver configs are updated.

This is correct and one of the things we discussed in the past and wanted to do is to avoid restarting all components on config changes and only restart components that changed (plus components that are earlier in the pipeline since the chaining model requires it). This would be another great addition to the Collector.

[jpkrohling]

@newly12, as @tigrannajaryan pointed out, this is part of a bigger discussion. I encourage you to join the collector SIG calls if you haven't yet (I missed the last few ones myself). I'm closing this PR, but this should not discourage you from contributing features that will lead to the outcome you wanted when you proposed this.