Thread 'tokio-runtime-worker' panicked at 'Receiver::next_message called after None'

[fgimenez]

A node running v0.8.22 is eventually crashing with the error in the issue title, full log here https://paste.ubuntu.com/p/KNd7cHfRZ6/

[mxinden]

8: sp_consensus::import_queue::buffered_link::BufferedLinkReceiver<B>::poll_actions

panicked at 'Receiver::next_message called after None'

From a first look BufferedLinkReceiver should either take a Fused receiver or never poll again once poll_next returned Poll::Ready(None).

substrate/primitives/consensus/common/src/import_queue/buffered_link.rs

Lines 142 to 146 in 38a974c

	let msg = if let Poll::Ready(Some(msg)) = Stream::poll_next(Pin::new(&mut self.rx), cx) {
	msg
	} else {
	break
	};

I am not familiar enough with the code to judge whether it should be impossible that the receiver returns Poll::Ready(None).

[tomaka]

What is worrisome is that this can only happen if the block import task shuts down, and this shut down is not supposed to happen.

[tomaka]

To give some context about this code: the BasicQueue::new function accepts a impl sp_core::traits::SpawnNamed as parameter, which it uses to spawn the actual import queue, which then runs in the background.

The BasicQueue struct completely hides any detail of this background task. It is BasicQueue that holds the channels that communicate with the background queue. The only way to send work to the queue is to use methods of the BasicQueue, and the only way for poll_actions to be reached is through BasicQueue as well.

The background task is designed to gracefully shut down if one of the channels is closed. Again, these channels are fields in the BasicQueue. If a channel is closed, that means the BasicQueue is dropped, and poll_actions shouldn't be reached.

Consequently, the only ways this issue can happen is:

• If the background task panics, but we would see a backtrace I suppose?
• If the executor drops the future for no reason, but I don't see why it would do that?

[arkpar]

It could be that the process was shut down by the OOM killer.
@fgimenez is there a record in /var/log/messages around that time?

[fgimenez]

@arkpar no messages around that time, in general the memory usage looks pretty sane compared to the total memory of the node (16gb), you can see in the image attached the memory usage in the last 6 hours, with the crashes happening always below 2GiB

[tomaka]

We merged #6876, which doesn't fix the root of the problem but will change the error to a different one (that might be more useful) for the next time.