[core/swarm] Graceful shutdown for connections, networks and swarms.

[romanb]

This is a proposal (at least for discussion) of enhanced graceful shutdown options for Connections, Networks and Swarms that builds on #1619. Even if #1619 is ultimately not merged, this may at least be a thread for discussing the topic of graceful shutdowns and its usefulness for libp2p. For a proper diff against #1619 see here.

Motivation

In the context of network I/O, a graceful shutdown is a form of shutdown that allows ongoing network traffic to cease in a controlled manner and at chosen boundaries (e.g. request/response boundaries), while at the same time rejecting new I/O streams (e.g. connections or substreams). The main purpose of a graceful shutdown of a networked application is thereby to minimise friction on restarts, incremental rollouts of updates, etc. that is, to minimise the amount of failed requests, responses or any errors resulting from sudden interruption of the ongoing network I/O at an arbitrary point. The more often networked nodes get restarted or redeployed, possibly as a result of frequent updates, the more errors and "hiccups" can result in the network traffic without graceful shutdowns. Graceful shutdowns usually also include redirection of traffic away from the nodes being shut down, which can sometimes be assisted by proxies or load balancers operating at network layers for more classical client-server deployments, but only the end-to-end applications and protocols typically know what needs to be done for a clean shutdown that avoids unnecessary errors.

How it works

This PR builds on the ability to wait for connection shutdown to complete, introduced in #1619 and extends the ability for performing graceful shutdowns in the following ways:

1. The ConnectionHandler (and thus also ProtocolsHandler) can participate in the shutdown, via new poll_close methods. The muxer and underlying transport connection only starts closing once the connection handler signals readiness to do so. The default implementations of ConnectionHandler::poll_close always signal readiness to immediately proceed with the transport-level shutdown.

2. A Network can be gracefully shut down via Network::close or Network::start_close, which involves a graceful shutdown of the underlying connection Pool. The Pool in turn proceeds with a shutdown by rejecting new connections while draining established connections. Draining established connections just means waiting for them to close, i.e. waiting for ConnectionHandler::poll_close followed by Muxer::poll_close to complete.

3. A Swarm can be gracefully shut down via Swarm::close or Swarm::start_close, which involves a graceful shutdown of the underlying Network in tandem with the NetworkBehaviour until the Network is closed and the NetworkBehaviour ready for shutdown.

Important Details

• Analogous to new inbound and outbound connections during shutdown, while a single connection is shutting down, it rejects new inbound substreams and, by the return type of ConnectionHandler::poll_close, no new outbound substreams can be requested.

• The NodeHandlerWrapper managing the ProtocolsHandler always waits for already ongoing inbound and outbound substream upgrades to complete. Since the NodeHandlerWrapper is a ConnectionHandler, the previous point applies w.r.t. new inbound and outbound substreams.

• While the connection handler is closing, it can still emit and receive events (e.g. to/from a NetworkBehaviour), but as mentioned above no longer receives new inbound substreams and can no longer request new outbound substreams.

• When the connection_keep_alive expires, a graceful shutdown is initiated rather than a sudden close via returning a keep-alive "error" as is done now.

• Connection::poll, Network::poll and Swarm::poll now return the event in an Option, since None signals termination as a result of a clean shutdown, i.e. in the manner of a Stream that is exhausted.

• I added ConnectionHandlerEvent::Close to allow a ConnectionHandler to request a graceful close of the connection (as opposed to returning an error, which will close the connection without delay).

• I added NetworkBehaviourAction::CloseConnection and NetworkBehaviourAction::DisconnectPeer to allow a NetworkBehaviour to request a connection to be closed (gracefully) or for a peer to be disconnected (immediately, i.e. not gracefully), respectively.

Usage Example

For an example of how a connection handler can make use of the ability to participate in graceful connection shutdown, I've included an implementation of ProtocolsHandler::poll_close for the RequestResponseHandler in libp2p-request-response. In the case of libp2p-request-response, these changes amount to allowing already ongoing requests to complete when a connection closes, while redirecting new requests to other connections.

Tests

I have essentially changed all tests that would previously emit frequent "broken pipe" errors (more of these since #1619 no longer always tries a clean connection shutdown automatically in the background task - only if actually requested) to perform a clean shutdown of the Swarms or Networks used in these tests.

[twittner]

Note that dropping a substream can cause write errors on the remote side, e.g. sending of initial data could fail when the substream is reset.

[romanb]

Yes, the current idea is to treat new inbound substreams analogously (as much as possible anyway) to new connections during shutdown - that is, to refuse them at the earliest possible moment. Any failures relating to attempts at creating new substreams or connections towards the peer that is shutting down should ideally result in retries with a different peer, much like without the graceful shutdown.

[twittner]

I am not sure if the shutdown logic as currently implemented can really alleviate error rates because any error in one substream will often lead to a connection close, causing failures in other substreams. For example, the request-response handler will immediately close the connection when any inbound or outbound upgrade fails other than via timeout or protocol mismatch. This means that if the remote closes the connection and my next outbound request over this connection fails, all my ongoing requests to this connection will fail too.

[romanb]

That is true, unfortunately. It depends very much on how substream write errors are handled, in the specific case of libp2p-request-response by RequestResponseCodec::write_request, which is currently completely up to the user. As you say, most other protocols probably consider write errors on a substream as fatal as well. The only somewhat clean solution that comes to my mind would seem to lead back to the ability for the substream multiplexers to have a configurable inbound substream limit similar to that of the connection limit for the Pool which can be changed during shutdown (i.e. set to 0), i.e. a limit that does not affect existing substreams but can be used to exert back-pressure in a way such that the remote knows it reached the current substream limit on that connection, but the connection itself is not broken.

[romanb]

So apart from the technical details concerning how it is done, I guess the main point for discussion here is whether the perceived benefits of providing such shutdown options justify the added complexity. I happen to think that, while the diff is large, the actual changes are relatively simple in nature, and I find that providing "clean" shutdown options for a Network or Swarm, analogous to a clean connection shutdown, is quite useful. However, as @twittner also pointed out, in many situations it may be sufficient to resort to means external to the libp2p-executing process itself to achieve similar effects for more graceful rolling updates of networked nodes, e.g. DNS. Not to forget that a graceful shutdown may prolong the time until the node is restarted and ready to receive new connections again, i.e. it may appear "unavailable" for a longer time than if it were to just drop all connections immediately and restart. One may of course also just take the view that any kind of graceful shutdown is completely unnecessary and trying to avoid errors during node upgrades not worth any effort, even with frequent node upgrades accompanied by process restarts.

[romanb]

There are some unresolved but important considerations relating to the handling of new inbound substreams and substream write errors during shutdown, see this discussion, which is why I'm putting this effort on ice for the moment.

[romanb]

Closing, since I don't think I will pick this up again. The main difficulty here is that a generic approach to a clean shutdown probably cannot be implemented without making too many assumptions about how concrete applications operate with the substreams on a connection and hence only a concrete application knows what can and needs to be done to gracefully cease traffic for a shutdown.