Support in-memory HTTP connections for scraping #3602
Conversation
This adds an in-memory HTTP traffic listener so components which communicate to other components through HTTP (e.g., for scraping Prometheus metrics endpoints) do not need to use the network. This is required to allow prometheus.scrape to continue to collect metrics from prometheus.exporter.* components if the HTTP server is protected by mTLS; otherwise, prometheus.scrape will not work until configured with support for mTLS. Closes grafana#2984.
This updates prometheus.scrape to use the custom dialer passed to it for opening connections to targets. This allows prometheus.scrape to properly scrape metrics from the in-memory HTTP listener.
|
We previously added this to the upgrade guide do we want to do the same here? |
I don't think so, before we documented it because we removed a field in integrations relevant to self-scraping over the network, but there wasn't anything for that here. This should be a fully transparent change to Flow users, and sets us up for the future where we have TLS support. |
| The exported targets will use the configured [in-memory traffic][] address | ||
| specified by the [run command][]. |
There was a problem hiding this comment.
I'm not sure why we document this phrase on the exporters.
Isn't prometheus.scrape the one initiating the connection towards the targets via the in-memory address?
There was a problem hiding this comment.
My thought with documenting it was to make sure that people knew it was a special target and not something they could scrape from outside of the agent.
There was a problem hiding this comment.
It threw me a bit off as the fact that the target is even a network-based thing was an implementation detail that the user didn't need to know, but I get it from that perspective.
Up to you whether you want it or not, it's okay by me.
There was a problem hiding this comment.
Right now my gut is saying we should leave this in (at least for now), since it's bordering between being an implementation detail and being something users need to know about, especially if agent.internal happens to be a real address on the user's DNS servers.
Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com>
* flow: support in-memory HTTP traffic This adds an in-memory HTTP traffic listener so components which communicate to other components through HTTP (e.g., for scraping Prometheus metrics endpoints) do not need to use the network. This is required to allow prometheus.scrape to continue to collect metrics from prometheus.exporter.* components if the HTTP server is protected by mTLS; otherwise, prometheus.scrape will not work until configured with support for mTLS. Closes #2984. * prometheus.scrape: use custom dialer for opening conns to targets This updates prometheus.scrape to use the custom dialer passed to it for opening connections to targets. This allows prometheus.scrape to properly scrape metrics from the in-memory HTTP listener. * docs: document in-memory traffic in Flow * misc: add CHANGELOG entry for in-memory HTTP traffic Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com>
* flow: support in-memory HTTP traffic This adds an in-memory HTTP traffic listener so components which communicate to other components through HTTP (e.g., for scraping Prometheus metrics endpoints) do not need to use the network. This is required to allow prometheus.scrape to continue to collect metrics from prometheus.exporter.* components if the HTTP server is protected by mTLS; otherwise, prometheus.scrape will not work until configured with support for mTLS. Closes #2984. * prometheus.scrape: use custom dialer for opening conns to targets This updates prometheus.scrape to use the custom dialer passed to it for opening connections to targets. This allows prometheus.scrape to properly scrape metrics from the in-memory HTTP listener. * docs: document in-memory traffic in Flow * misc: add CHANGELOG entry for in-memory HTTP traffic Co-authored-by: Paschalis Tsilias <tpaschalis@users.noreply.github.com>
github-actions
bot
added
the
frozen-due-to-age
This PR integrates github.com/rfratto/ckit/memconn into Flow, which was already in use in the integrations-next subsystem in static mode.
memconn allows establishing fully in-memory HTTP connections to avoid the network stack. This is primarily useful in the presence of mTLS, where otherwise Flow would need to know how to connect to itself as a client with the proper TLS certificates.
As of this PR, in-memory HTTP connections are only used for prometheus.exporter.* components and prometheus.scrape, where prometheus.exporter.* components export an in-memory address, and prometheus.scrape can use a custom dialer to connect to that address.
I am not really happy with the API as-is, but @mattdurham is actively working on refactoring the controller/module API which would be able to implement this logic in a cleaner way; so for now I think the slightly ugly API is acceptable.
Closes #2984.
Related to #2715, grafana/alloy#456.