[AIP-49] Breeze OpenTelemetry Integration

[ferruzzi]

This PR builds on the recently-merged Breeze StatsD Integration PR.

#29449 established a Docker Compose file which launched everything needed to emit metrics over StatsD and view them in Grafana. This PR lays the groundwork for the same process but uses OpenTelemetry instead of StatsD. It will be a required test environment for upcoming work on AIP-49 OpenTelemetry Support for Apache Airflow.

I included a disclaimer in the documentation page that the underlying work to make this useful is pending.

The first commit b84e96c makes some adjustments to the previous StatsD PR in order to make them more reusable. Second commit 8092112 implements the new work.

cc @o-nikolas @vincbeck @syedahsn @vandonr-amz

[vincbeck]

I would not qualify myself as a Docker expert so take my approval lightly but I did not see anything standing out

[o-nikolas]

Are these statsd values being set temporary? Since this is the OTEL integration I would expect to see OTEL env vars here not statsd

[ferruzzi]

Correct, at the moment there is no code in Airflow that would accept other values so I left these for now, but in hindsight that doesn't make any sense I guess. I'll remove them and leave a note to see the statsd one for formatting once they are implemented

[ferruzzi]

Sorry for the force, I had to rebase

[ferruzzi]

Main is broken again, should have not rebased yet I guess.

[potiuk]

looks transient error with httpbin. rerunning.

[potiuk]

woooho.. progress.

[ferruzzi]

Cool. I peeked in the PRs list and there were a LOT of them failing so I assumed it was another main hiccup.

[potiuk]

Actually - if my guess is right, then we have to fix it and NOT use httpbin in our tests - this means that they rely on external service we have no control over and SOMEONE my deliberately actively impair our CI.

[ferruzzi]

FAILED tests/providers/http/hooks/test_http.py::test_do_api_call_async_retryable_error - aiohttp.client_exceptions.ClientConnectionError: Connection refused: GET http://httpbin.org/non_existent_endpoint

I didn't copy the error message from before, but it looks familiar, I think it's the same transient one @potiuk mentioned.

[potiuk]

FAILED tests/providers/http/hooks/test_http.py::test_do_api_call_async_retryable_error - aiohttp.client_exceptions.ClientConnectionError: Connection refused: GET http://httpbin.org/non_existent_endpoint

I didn't copy the error message from before, but it looks familiar, I think it's the same transient one @potiuk mentioned.

I tihnk this is simply some kind of instability of the http://httpbin.org/ - or maybe they (or their systems) THINK that we are DDOS-ing them with our CI and block the attempt if there are too many.

With Airflow's scale of PRs and CI it's actually even likely :D

[ferruzzi]

It just passed, so unless someone fixed something then it's intermittent. 👍