[Tempo] Improve vision on dropped spans

[joe-elliott]

Currently observability on dropped spans is poor. The agent is based on the OTEL collector which has this issue:

open-telemetry/opentelemetry-collector#2434

Additionally if batches are dropped b/c the queue is full they go unmetriced. This is logged, but not metriced as a failed send: "Dropping data because sending_queue is full"

https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/monitoring.md#secondary-monitoring

Let's keep an eye on these issues and revendor when they are fixed. If we have spare cycles these are also good issues for upstream contributions.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.