Messaging / Chat

[JeremyOne]

I have been working on an idea for a messaging app that works 100% inside the IPFS network, and am looking for input from people that know more about IPFS than me.

Here is the basic idea:

1. Create a web app using only static files (html, js, css...), which would be available to download over IPFS and run in a browser.
2. The app would use 100% IPFS API calls to store and retrieve data on network, and would have no other coordinating (non-IPFS) server.

Here is my big question:
What is the best way to pass a message from one user to another?

My initial idea is, for each user, create a "queue" of outbound messages (a json list) that is linked to a IPNS name. When a message is "sent" it is added to this list, published to IPFS and linked to the IPNS name. During a conversation, each participant would then be actively checking the other participant's queues for updates. When a message is found by the receiver, they keep a "message received" entry to their outbound queue until the sender removes the original message from their queue.

This seems like it would work, but there would be lots of polling for updates on both ends of the conversation. Maybe someone can think of a better method?

[davidar]

See ipfs/notes#42

Edit: also ipfs/notes#40

[jbenet]

Good steps!

Take a look also at CRDTs and attempts to make messengers/forums on them. We can map that trivially.

one important thing to consider is signatures, to ensure data is correct.

This seems like it would work, but there would be lots of polling for updates on both ends of the conversation. Maybe someone can think of a better method?

we'll have pub/sub to make this way better.

[amstocker]

I would be interested in forking scuttlebot to use ipfs as a backend.

[whyrusleeping]

@dominictarr what would be involved in that process?

[JeremyOne]

I read up on the pubsub feature, and it sounds prefect for messaging, but I was not clear on the status, sounds like it is under development?

[whyrusleeping]

@JeremyOne ipfs pubsub is not written yet. It has been discussed and we have a few different ideas about its implementation, but I dont think its going to be ready for a little while due to our limited dev bandwidth

[amstocker]

I think for the meantime we could probably use scuttlebots functionality on ipfs with not too much effort hopefully. I havent browsed the source but I've read their docs and I think we need to (1) make a backend plugin for ipfs and (2) make it so objects arent passed around just their hashes and some metadata (then we would use ipfs to transport the actual message contents).

[JeremyOne]

SSB looks interesting as well

[amstocker]

relevant: ipfs/notes#64

[jbenet]

(haha! i was hanging out with @dominictarr when i saw this thread.)

I think what makes sense here is to look at SSB's component protocols and
grab the pieces that are necessary -- like, you may not need full SSB as
IPFS already has secure transports-- maybe what you want most is the
scuttlebutt 2-degrees gossip replication strategy, and maybe you want to
layer the nice hex UI on top too. -- though @dominictarr can probably tell
you better.

Also, this ties really well into some stuff we've been working on with
pinsets, and thinking through ipfs-cluster. It may be that an
ipfs-scuttlebot module would layer over the core of ipfs and produce a
dynamic pinset based on the 2-degrees thing. (the easiest way to implement
this is to add a recursive pin on the head of a "follow-list" -- where an
entry in the follow list has to be expressive enough (dag path notation?)
to traverse 1 hop, but not all).

(separately, you can indeed mount all of SSB on top of IPFS directly,
mounting it onto IPFS's libp2p node, but it's a less integrated approach
and i think that since both use merkledags, they should be integrated
better)

[dominictarr]

You could certainly distribute ssb's data over ipfs, but it would involve a lot of polling, more or less as @JeremyOne describes. Also, ssb replicates data in order (posts are organized into feeds which are append only, and read in that order.

This is a much more constrained structure than ipfs, but this allows us to optimize replication for this structure - also, since feeds are replicated over a gossip network, it's pretty likely that two replicating peers will have feeds of interest to each other (because of mutual friends).

Also, by replicating in the same order as messages where created, it means if replication fails, you are still in a valid state, which gives quite predictable properties as a database.

There isn't really a general purpose replication system, heavy tradeoffs in all directions.

So for that reason, I think you'd be better off running ssb alongside ipfs, rather than trying to run one on top of the other. And, i bet there are other protocols that solve other problems better, and so the secure p2p internet ("cypherspace") will be built upon a variety of decentralized data structures.

[jbenet]

but it would involve a lot of polling

not with pub/sub, right?

This is a much more constrained structure than ipfs, but this allows us to optimize replication for this structure - also, since feeds are replicated over a gossip network, it's pretty likely that two replicating peers will have feeds of interest to each other (because of mutual friends).

i think this can be expressed in a path notation. could you point me to the replication algo of ssb? I bet i can turn it into a dag path string. If I do, then this:

There isn't really a general purpose replication system, heavy tradeoffs in all directions.

is called into question.

I suspect that proving it false is doable, and likely already exists in the literature already. There is of course, one usual answer that is readily correct: provide sane defaults, and leave the special cases up to the application with a simple, clear language (in our case, one string).

So for that reason, I think you'd be better off running ssb alongside ipfs, rather than trying to run one on top of the other. And, i bet there are other protocols that solve other problems better, and so the secure p2p internet ("cypherspace") will be built upon a variety of decentralized data structures.

What you're suggesting is analogous to saying that computers should have all link layer 2 interfaces, instead of a common network protocol on top of the individual network. IP does not solve {Ethernet, WiFi, ...} concerns... and yet...

Many decades ago, there were many different networks, with protocols suited to the specific, individual use case. Computers were in isolated networks. Then JCR Licklider had the idea to make the "Intergalactic Network", to combine them and make them all mutually useful. Today you use the Internet, and you use IP from every network. Some decades after, there were many hypertext webs, and one came to glue them all together with one simple Universal addressing scheme, to make a single Web. History sure does rhyme.

And this is precisely what IPFS is about. IPFS is not about choosing one p2p system over another, but about integrating them. And the claim this is "not true" ignores history, and ignores what we've set out to do, and what we've already demonstrated.

---

We've been arguing about this for months and i would like to resolve this disagreement concretely, with code and/or proofs. So, give a fair shot: list the concerns you have -- what do you think cannot be done? If i show it can-- you should honestly reevaluate the claims. So far, i can list:

• demonstrate a general purpose replication algorithm
• prove that it captures SSB replication
• prove that custom protocols can be mounted (this exists today)
• prove custom applications can govern their replication + use of content.

Are there others?

[davidar]

IPFS is not about choosing one p2p system over another, but about integrating them.

I think this point needs to be publicised more, as for many people the first question is "why should I believe that IPFS can do any better than all these other P2P systems that have been around for years?"

See https://news.ycombinator.com/item?id=10330475 for example

[davidar]

• http://cr.yp.to/im2000.html
• https://news.ycombinator.com/item?id=10406362

If mail storage is the sender's responsibility, then perhaps senders should be allowed to seed their outgoing messages directly from their personal and home devices, and rely on others who have included their identity in their web-of-trust\address-book to propagate notifications of changes to their outbox, without relying on their ISP to do so.

If the sender did not have at least one personal device connected to the internet 24/7, then they can perhaps purchase storage and bandwidth from a competitive online market, similar to the seedbox market for torrents, without surrendering their online identity or address to a commercial entity.

[jbenet]

@davidar interesting model. I signed up for the im2000 ml, but haven't seen much develop yet. worth discussing all this with djb

[davidar]

@jbenet Yeah, I think it was one of those ideas that never really went anywhere. Would be cool if we could resurrect it :)

Edit: As the name suggests, it was first proposed 15 years ago

[victorb]

Did something simple regarding this: https://github.com/victorbjelkholm/ipfschat

Very alpha at the moment but should work as advertised. Is more like a global chat with everyone in the same "room" but can easily be improved. Should I continue working on this?

[haadcode]

We realized in the weekly call that this issue exists and hasn't been updated in a long time. So here we go: Orbit - serverless p2p chat on IPFS https://github.com/haadcode/orbit. Take a look and join the development :)

[beenotung]

I'm working on similar application

As lone as the "connection" is built between user (each knows other's ipns hash), it's possible to exchange messages

But the first question is how can I know my friend's ipns hash?

when we say 100% only depends on ipfs, i.e.

1. cannot use another server to let the user to search another user by name, phone number, email, e.t.c.
2. cannot even use email to send your ipns hash to another user, what if the email (e.g. Gmail) is blocked by the receiver's ISP? (e.g. mainland china)

[KRYPTOHAUS]

The feeds provided by secure schuttlebutt would still be pretty useful, despite the work being done on the chat front; feed functionality is core to social networking-style apps/dapps.

[victorb]

@beenotung you could make use of pubsub and have a channel where everyone publishes their username/peer-id and show that in the UI, letting users to confirm the ID out-of-band and "add them" somehow...

[mitar]

Just recently I developed a blockchain on top of IPFS using its pubsub feature. It is really easy to create an append only log using IPFS and pubsub. I think it would be really rad if scuttlebot would use IPFS to distribute its log, and then just provide ways to discover pub/sub endpoints and merge objects from the append only log, as it is already doing now. But there is really no need, from my understanding, to replicate all the networking stuff.

Do not forget, IPFS is not just for storing files, but contains objects and links between them. I mean:

{
  "key": "%vhP8tyeB+7cVLbIHd6wEd36AVEcUsZgwTYigpcx6Qn0=.sha256",
  "value": {
    "previous": "%NA/4By9K3L0OmVS2eD8le05uUW94ukDNbX16V3ZApi8=.sha256",
    "author": "@hxGxqPrplLjRG2vtjQL87abX4QKqeLgCwQpS730nNwE=.ed25519",
    "sequence": 2,
    "timestamp": 1439158769923,
    "hash": "sha256",
    "content": {
      "type": "post",
      "text": "Hello, world!"
    },
    "signature": "EQngCchOejwfWAcZ2Xgr5QR6iquBQVlF++1/ZOLlRJQfyj4TxHk6MHRUKV/o7L35h2zfL1K+Il991JxrxCT+BA==.sig.ed25519"
  }
}

Looks so similar to my blockchain (pseudo) objects, slightly edited:

{
    "Data": {
        "Luck": 0.9365737341333908,
        "Proof": {
            "Quote": "8uS51PPifMPSZset3nCdM4MSF7RrXmHCwqn7hYxeD3KdyUoV7iJBqBVaUgT4kMyR6U65mKvcQH6MJxAYCcbZKVGyVjbd",
            "Attestation": "3oF4jk"
        },
        "Time": "2017-03-18T07:52:17.981Z",
        "MinerId": "QmUyo4YPVQTbNnXk7CUMmQquFxsa9DPcXautycgzxs13MR",
        "Sequence": 27560
    },
    "Links": [
        {
            "Name": "parent",
                "Hash": "Qmd99fbSWUgwp2KokjqG1Kka5aDEZurjYjY2jibuS9k5Y9",
                "Size": 21489393
            }
        }
    ],
    "Hash": "QmQoRL9UdgMobfeAy1uUg7Xo6nT67GdMRbVjwhrq6pqjKH"
}

Only signatures are missing, but IPFS also supports keys (TODO: #120).

And then linking to a large file or movie from a scuttlebot object would be really easy.

Personally, this is how we can really achieve decentralized web again, by stacking layers.

[joantune]

@mitar interesting! both the work and the blog post. [Off Issue]: I sent you an Ask on Tumblr, you ought to contact me [I think you can through the Ask thing, it was tricky to put my email there for some reason it seems that yahoo is better at protecting against links in messages than password theft 😛 .. ], I can put you in touch with that guy for the decentralized computation part. I'm sure he's interested at the very least to learn about it. And now that the has a viable business he might be willing to give back, and especially it would help with making his initial vision come true

[matthiasbeyer]

The scuttlebutt "append only log" sounds a lot like a thing that could be done with IPFS and IPLD in combination, right?

Also, if each post links to the profile which punished it, one gets multi-device support form free (given that the private key for signing is shared between the devices), or am I mistaken at this?

Using IPNS, one can update a profile (which creates a new entry in IPFS) without breaking "contact information".

For me, it looks like IPFS+IPNS+IPLD can be used to build a social network. Scuttlebutt seems like the first step, porting it to IPFS would be a leap forward.

[whyrusleeping]

@matthiasbeyer I would love to port scuttlebutt to ipfs. I was actually just thinking about that earlier today, since I recently got patchwork installed and have been really impressed by it.

If anyone is interested in working on that, i would love to help out.

cc @diasdavid and @dominictarr

[beenotung]

@victorbjelkholm I tried the IPFS pubsub, it seems to rely on WEB RTC seed node which further relies on DNS lookup

Here is the example I made: https://ipfs-chatroom.surge.sh/
It failed some time ago because "the list of seed node is outdated", it is fixed after updating the library (with hardcoded seed node I think).

Is there a way to completely not rely on the current centralized/decentralized but not p2p network system? Something like NDN but do not require massive deploy of NDN router?

[mitar]

@whyrusleeping, scuttlebutt is now issuing grants. I think it would be awesome if maybe somebody would apply to port it on top of IPFS. I think this would be an amazing stack. IPFS below, scuttlebutt over it, and than patchwork over that. In this way it would be even more censorship resistant.

scuttlebutt hubs would then be mostly IPFS peers which are online regularly and proactively pull scuttlebutt's IPFS data to replicate it.

This seems also maybe related: https://github.com/noffle/ipfs-hyperlog

[haadcode]

There's an append-only log data structure implementation that is fully on IPFS and has been used successfully for a long time in OrbitDB: ipfs-log.

With an upcoming release, we finally support write-permissions which means that we can have per-user (key) feeds as well as multi-user feeds (allow multiple keys to write to one feed). To have feeds from which entries can be deleted (from the database, not from the log), one could use OrbitDB's feed database. The log itself is just a data structure on IPFS and can be used with or without Pubsub or IPNS (that's essentially what OrbitDB does with the log, handle the coordination).

I wonder if Scuttlebut could use ipfs-log to get on IPFS easily? Or OrbitDB's eventlog (which is the same as the append-only log but with coordination through Pubsub)?

The data structure seems to contain all the same elements, this is what an entry looks like in ipfs-log (which then gets saved to IPFS as an object):

{ 
    id: 'C',
    payload: 'four',
    next: [ 
      'QmeLqJ7EotieQY2EdY1gzCpot8LdDC6xkjAgnD6J8jV7tF',
      'QmZyh41WKjj61oSSaNfEgK8Ejy5ufBuwZRzbFcwVnsViwS' 
    ],
    v: 0,
    clock: { 
      id: 'C', 
      time: 3 
    },
    sig: '3044022020866efcc4b63804723cc9e3263e80ce830f81a6a2f963e4c6318b553b71f39002202214c7f8af2d2e0f975eda557e595f25b5c933b3a2e82ae38e441b386a13117a',
    key: '04eb9b386bc841f2172fe912c3a3384bb24983a3de8bba27b0a4c87b8bd12f6c5b28f1965c541364757e317d3df942071fd274bcb989b76f0ad06acbc027d72dfa' 
}

I'd be happy to help if anyone wants to take a closer look at ^ and see if we can use it.

[matthiasbeyer]

These data structures could indeed help a lot with the problems I wrote about in my blueprint of a distributed IPFS social network!

[daviddias]

Just wanted to leave a pointer here that with the y-ipfs-connector, you can also grab the chat example from yjs at http://y-js.org/#!/examples and build a chat over IPFS.

Here is a tutorial (video + text) on how to do it for a collaborative doc https://blog.ipfs.io/30-js-ipfs-crdts.md.

More docs from yjs https://github.com/y-js/yjs/wiki/Custom-Types

[beenotung]

For your reference, this demo use IPFS with y-js (and with typescript annotation)
https://github.com/beenotung/ipfs-chatroom.git