Request flood in web browser

[lidel]

Problem

I've been testing delegated routing in browser context, namely with js-ipfs embedded in Brave (ipfs/ipfs-companion#716).

It produces pretty intense list of pending requests when browsing /ipns/tr.wikipedia-on-ipfs.org 🎠 💦

Modern browsers limit the number of concurrent requests to ~6 per hostname.
Everything over the limit lands in "pending" state.

On some pages it takes a dozen of seconds for the browser to finish a pending list like the one above (it was multiple screens), and while it happens nothing else works.

When run in regular page, this flood of swarm connect results in slow browsing experience (nothing loads, "waiting for available socket" displayed in bottom left).

In my case (Brave + chrome.sockets), coupled with regular preload requests it produced more severe net::ERR_INSUFFICIENT_RESOURCES:

Fix

My initial idea would be to limit the number of concurrent requests using something like p-queue around these lines:

js-libp2p-delegated-content-routing/src/index.js

Lines 90 to 92 in 69f1f94

	const results = await Promise.all(
	addrs.map((addr) => {
	return this.swarm.connect(addr.toString()).catch(() => {})

I'll prepare PR with this.

[lidel]

Note to self:

• (while testing with public delegate servers) when we throttle the number of parallel requests DoS protection gets triggered at Nginx
• we don't cache successful swarm connect: requests are sent over and over again, but could cache it as there is an expectation that connection is alive (at least for 1minute..)
  • this will decrease the number of requests significantly, as swarm connect is the main offender

[lidel]

Update: throttling/caching helps, but is not enough. Some requests can take a very long time and will suffocate request queue because they have no built-in timeout (namely /api/v0/refs).
This is a separate issue: filled #15

[alanshaw]

@lidel they are mostly calls to swarm.connect - I'm not sure they are even needed. See the conversation here: #14 (comment)

[lidel]

@alanshaw yeah, that confirms what I'm seeing: swarm connect from delegate to js-ipfs always succeeds, no matter what multiaddr was used:

AFAIK autorelay feature is off on bootstrap nodes. However bootstraps in this module do not come from js-ipfs (param is undefined as js-ipfs does not pass bootstraps)

This means p2p-circuits are never used(!) however it works because node0.preload.ipfs.io is reusing already existing connection that my node established with it because its one of bootstrap nodes.

(go-ipfs ignores multiaddr if it already has a working connection with same peerId)

This means we could either remove swarm connect entirely (js-ipfs will maintain connection to delegate/bootstrap node, no need to force it), or finish early, after first success (decreasing number of requests from one per bootstrap, to a single one).

Given the docs explicitly state that delegate needs to also be a bootstrap node (libp2p/js-libp2p#371), I'd vote for removal of swarm connect call and relatec code.

[jacobheun]

Catching up on this now, sorry for the delay.

Given the docs explicitly state that delegate needs to also be a bootstrap node (libp2p/js-libp2p#371), I'd vote for removal of swarm connect call and relatec code.

Yes, we should remove this code. Right now we're telling the delegate to connect to us through every bootstrap as relays, which isn't going to work as they're not relays. I think this code may have made the assumption that we could be using a delegate that's not publicly dialable, which is wrong.

Getting the Priority Peer or (Friend Peer) functionality in place and documenting that users should have their delegate node as a priority peer should be sufficient for us just calling refs on the delegate.

All the bootstrap node stuff just needs to go away.

[lidel]

@jacobheun neat! I removed bootstrap node stuff as part of #16