Open Problem: PubSub at Scale (1M, 10M, 100M, 1B.. nodes) #5
Conversation
|
Do not forget to link to: |
First stab at Pubsub at scale. Will follow up with a few updates and references to related literature very shortly.
Version good to review.
|
@daviddias @vyzo : It would be great to get some thoughts and a review of this Open Problem on PubSub. |
There was a problem hiding this comment.
Thanks for pushing this update @yiannisbot! It is has a lot more shape now.
Overall I believe it needs:
- More concrete pointers to what is possible in the IPFS/libp2p ecosystem with details of what it actually does (rather than highlighting what is possible on the whole field)
- Highlight examples of use cases with some numbers for benchmarking (so that people understand the "scale" that we are talking about)
OPEN_PROBLEMS/PUBSUB_AT_SCALE.md
Outdated
|
|
||
| <strong> Gossip Protocols </strong> | ||
|
|
||
| Gossipsub is a gossip-based pubsub system developed within libp2p. Gossipsub borrows concepts from related literature (see list below) and blends them together to produce an efficient pubsub protocol. Traditionally, the concept behind gossiping is to address the issue of load-balancing between all nodes forwarding messages in the system. According to gossiping, once a node receives a message, it does not broadcast the message directly to all nodes subscribed to some topic, but instead it is choosing a fraction of nodes (we can use *t* to denote the number of nodes chosen) to distribute the message. In turn, those *t* nodes are choosing a further *t* nodes to distribute the message further. Clearly, receiving the message more than once is perfectly possible in gossiping systems. If a node receives a message twice, it discards the second (and any subsequent) message it receives. |
There was a problem hiding this comment.
This explanation needs to have a greater focus on what libp2p-gossipsub today is rather than the general case of gossiping in the literature
OPEN_PROBLEMS/PUBSUB_AT_SCALE.md
Outdated
|
|
||
| ##### peer-base collaboration messaging protocol (Dias Peer Set) | ||
|
|
||
| - https://github.com/peer-base/peer-base/blob/master/docs/PROTOCOL.md#peer-base---protocol-explanation |
There was a problem hiding this comment.
Needs quick explanation (1 paragraph)
There was a problem hiding this comment.
Is this a well-known protocol? Or is it/has it been used in libp2p? I can't see anything special about it and the RFP is getting long..
OPEN_PROBLEMS/PUBSUB_AT_SCALE.md
Outdated
|
|
||
| - We've collected a vast amount of Research Material around PubSub. It can be found at https://github.com/libp2p/research-pubsub | ||
|
|
||
| Related literature in the broader research community has worked towards two incarnations of pubsub systems: i) content-based pubsub and ii) topic-based pubsub. |
There was a problem hiding this comment.
There are more, for example type, concept and also the topic base can be expanded to tag base.
Here is a quick list of many of the systems I compiled earlier:
Subject/Topic/Tag based - Lists of subscribers per subject/topic/tag
- IP Multicast
- BMTP
- SRM Framework
- Scribe (on top of Pastry)
- Bayeux
- Rappel
- Tera
- SpiderCast
Content Based Subscriptions (apply filters/matching to forward messages)
- Pragmatic General Multicast
- Elvin
- SIENA
- Provides a Scalable and efficient pubsub - G. Banavar, T. Chandra, B. Mukherjee, J. Nagarajarao, R. E. Strom, and D. C. Sturman, “An efficient multicast protocol for content-based publish-subscribe systems,” presented at the Proceedings. 19th IEEE International Conference on Distributed Computing Systems, 1999, pp. 262–272.
- Gryphon
- JEDI
- Hermes
- Rebecca
- Meghdoot
- PHT
- O-P DHT
- PastryStrings
- Sub-2-Sub
Type based- middle ground approach between the two above
Concept/Semantics based - XML and semantic stuff everywhere and on the fly, more flexible but also more complex
There was a problem hiding this comment.
Included mention of most of these. Can add some survey papers.
|
|
||
| ### Known shortcommins of existing solutions | ||
| > What are the limitations on those solutions? | ||
|
|
There was a problem hiding this comment.
The known shortcomings is currently empty.
There was a problem hiding this comment.
One that should be highlighted is that almost no PubSub literature covers Tree Forming, they all mostly assume that the number of peers is consistent (no churn) and that they can dictate the spanning tree to their own design at any moment in time (without contemplating the challenges with having an accurate perspective of the network).
It would be valuable to highlight some papers that focus on efficient routing at the underlay level (i.e. OSPF, RIP, etc) to contrast and take inspiration from.
There was a problem hiding this comment.
One that should be highlighted is that almost no PubSub literature covers Tree Forming, they all mostly assume that the number of peers is consistent (no churn) and that they can dictate the spanning tree to their own design at any moment in time (without contemplating the challenges with having an accurate perspective of the network).
To my knowledge, this is mostly true for structured, managed and mostly cloud-based pub/sub systems. Some of this has now been discussed together with the differences to unstructured systems.
It would be valuable to highlight some papers that focus on efficient routing at the underlay level (i.e. OSPF, RIP, etc) to contrast and take inspiration from.
I didn't do that at this point. I don't have a clear view of how to link between routing in the underlay and overlay. E.g., run dijkstra on the overlay? Or take routing input from underlay protocols and optimise overlay routes?
OPEN_PROBLEMS/PUBSUB_AT_SCALE.md
Outdated
| - Social Network | ||
| - Chat | ||
| - Newschannel | ||
| - Video Broadcast |
There was a problem hiding this comment.
This needs to be expanded to inspire the reader to think about the scales that we are thinking about.
|
@JGAntunes This is Open Problem is at the core of your research -- ipfs/notes#266 --, would love your review and suggestions in what to include, specially all your insights from your recent literature review :) |
|
Following -- #5 (comment) @JGAntunes created a really nice table with a summary eval of multiple systems |
|
To clarify: I do not expect us to do a complete multi decade literature review in order to publish the first version of the open problem (we can always augment latter), however, it is important that we at least ship with a good representation of the multiple lines of work and don't focus too much in one. |
OPEN_PROBLEMS/PUBSUB_AT_SCALE.md
Outdated
|
|
||
| ## Long Description | ||
|
|
||
| Pubsub systems are a subgroup of broadcast trees, according to which, once messages are published, they are broadcast to all nodes in the tree. Although broadcast can ensure timely delivery of messages to all receipients, it causes severe stress to the system in terms of bandwidth needed to deliver the messages. |
There was a problem hiding this comment.
This is why I mentioned they are a "subgroup" :) but I rephrased in the new version. Thanks!
OPEN_PROBLEMS/PUBSUB_AT_SCALE.md
Outdated
| - Julien Gascon-Samson, Franz-Philippe Garcia, Bettina Kemme, Jörg Kienzle, Dynamoth: A Scalable Pub/Sub Middleware for Latency-Constrained Applications in the Cloud, IEEE ICDCS 2015 | ||
|
|
||
|
|
||
| ### Known shortcommins of existing solutions |
|
@yiannisbot let me know when you want a review again, I can put it on my todo for tomorrow :) |
|
@daviddias: This is now just out of the oven. Much more mature than before. Please have a look in case you spot something that needs attention, otherwise we can finalise. |
No description provided.