Stages - Exit Cleanup

[jackson-sandland]

@garious Thanks for the precedent set. I've made updates for every stage file except for the one's you've used in your example.

Running cargo fmt resulted in many formatted files unrelated to this work.
Didn't want to fold that into this PR.

What do you think about having a pre-commit hook that runs cargo fmt?

Addresses #232

[garious]

Thanks @jackson-sandland, this is the right idea, but will likely breaks things. We'll need to add tests similar to the ones in record_stage.rs to ensure only Disconnected errors cause the thread to exit. A channel timeout or TryRecvError::Empty, for example, should not cause the thread to exit. We'll need to check with @aeyakovenko, but I think the way some of these stages are implemented is to poll the channels for data with a timeout, poll the exit variable, and repeat. Once the exit variables are gone, we'll be able to safely block on those channels without timeouts.

[jackson-sandland]

@garious Aha, so these changes would allow the thread to exit on any err when it should only exit on a disconnect. I just read the discussion in Discord too. It sounds like the brakes are paused on exit cleanup. Feel free to decline this one :)

[garious]

@jackson-sandland, I'm going to close this PR for now. While I opened the ticket a while back, I don't think we understood the implications until the PR showed us the code. Since we're not able to merge it in the near-term, I updated the description to reference #232 so that when we revisit the ticket, we'll have an opportunity to leverage your work. Thanks for taking this on!