Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Increase channel buffer size from network worker to node tasks #6009

Closed
tomaka opened this issue May 13, 2020 · 4 comments · Fixed by #6064
Closed

Increase channel buffer size from network worker to node tasks #6009

tomaka opened this issue May 13, 2020 · 4 comments · Fixed by #6064
Labels
I9-optimisation An enhancement to provide better overall performance in terms of time-to-completion for a task.

Comments

@tomaka
Copy link
Contributor

tomaka commented May 13, 2020

Right now the buffer size is 5 elements. In other words, the worker can send 5 elements to a libp2p tasks, after which it has to go to sleep waiting for the libp2p task to make some space in that queue.

In practice, however, grandpa seems to sometimes send bursts of hundreds of messages, and it probably wouldn't hurt to increase this buffer size to a bit more than 5.

This requires changes in libp2p to make this buffer size configurable, but I'm opening it here to not forget to update Substrate later.

@tomaka tomaka added the I9-optimisation An enhancement to provide better overall performance in terms of time-to-completion for a task. label May 13, 2020
@tomaka
Copy link
Contributor Author

tomaka commented May 19, 2020

It seems that this change (#6064) is indeed positive.

We deployed around 11:37.
CPU usage of tasks (red is the network worker):

Screenshot from 2020-05-19 12-13-15

Size of queue of messages to send out:

Screenshot from 2020-05-19 12-14-33

These changes indeed seem to indicate that the network worker is often waiting for some node task to be ready, and that during this wait there's an accumulation of messages in the buffers, leading to the network worker taking a long time to process them afterwards.

@tomaka
Copy link
Contributor Author

tomaka commented May 19, 2020

I'm going to reopen this, because it seems to be that these channel sizes are still not as large as they should be.

Before blindly increasing the sizes again, we should introduce some way to know when the network worker is waiting for a libp2p node.

@tomaka tomaka reopened this May 19, 2020
@tomaka
Copy link
Contributor Author

tomaka commented May 19, 2020

There has also definitely been an improvement to these spikes of networker worker CPU work.

Screenshot from 2020-05-19 13-47-34

@tomaka
Copy link
Contributor Author

tomaka commented Jul 29, 2020

No longer needed after #6692

@tomaka tomaka closed this as completed Jul 29, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
I9-optimisation An enhancement to provide better overall performance in terms of time-to-completion for a task.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant