Bug 1828066 - At shutdown block with a timeout and bail out if that fails.

[badboy]

No description provided.

[badboy]

What happens in the bug?

Crash report: https://crash-stats.mozilla.org/report/index/47e42dbc-f99c-474f-8154-bbf470230413#allthreads

main thread hangs in glean_core::dispatcher::global::block_on_queue() during shutdown (code).

thread glean.init hangs in dispatcher::flush_init() (code)
Assumption: some task scheduled a ping?

thread glean.dispatcher triggered handle_client_active(), thus submitting a ping synchronously.
This hangs in std::sys::windows::fs::rename().

thread glean.upload is waiting for viaduct to make a request (why?)

This fix

We only briefly wait for the dispatcher queue to empty. If it currently is running a task that takes too long, we just skip any further shutdown cleanup.

Guarantees?
We only wait up to 5s on the dispatcher.
If the dispatcher unblocks quicker we continue.
However we then shut down the dispatcher, which blocks again. In theory there's a window where new tasks get put on the dispatcher and could block for longer.
We also keep the blocking on the uploader shutdown (up to 30s), as well as persisting ping lifetime data (itself I/O, so could block).
So really we do catch this one instance where somehow a ping submission is in progress when shutting down, but taking long.

The 5s timeout is currently hardcoded. Should we make this compile-time/runtime configurable?

[chutten]

Since we've recently learned the benefit of knowing timeouts within wait timing distributions, perhaps include a note that we expect most samples to be less than the default timeout of 10s