Migrate sketch updates to use CRDTs

[catarak]

Nature of issue?

• New feature request

New feature details:

As suggested in the discussion in #1337 and through issues such as #1213, I think many saving issues will be alleviated by switching to dispatch changes to sketches rather than replacing the entire contents on the sketch.

[catarak]

It seems like the way to go with CodeMirror is to use y.js

[vulongphan]

Hi @catarak, could you please help me navigate which part of the code is responsible for saving changes in a sketch?
Thank you!

[MohdImran001]

Hey @catarak , do you have any suggestions how we could approach this issue. I have decided to use y.js web sockets provider for synching client and server but I didn't find any provider for saving data into mongodb. Maybe we have to handle it ourselves on the server. I would love to know your suggestions on this.

[MohdImran001]

Hi @catarak, I am working on it and I have decided to take the following approach:-

I am using the y-websockets package to sync sketches across server and client. The y-websockets also allow us to sync the content across different tabs using cross-tab communication and it will help us to also implement this feature #1249.

On the client side, the users first needs to save the sketch for CRDT to work to prevent them from saving junk sketches. Once the sketch is saved, the y.js will sync the changes across the server.

On the server, we could use the levelDB y.js plugin to persist the changes coming from the client. When the server receives updates, we will store them in levelDB and will update the changes of the sketch in MongoDB.

I have decided to use the Map Data Structure to sync changes. Will use files id's as keys and their content as values. If a file hasn't been updated its content in the map will be empty and we will not update this file. This will allow us to update only those files which have updates rather than saving whole sketches.

I am working on an initial setup which we can easily extend with Webrtc to implement features such as #1337 and #1651. I would love to hear your feedback on this approach. Can I connect with you on email to discuss some doubts ?

[jgrizou]

Not entirely sure this is linked but I developed a small webapp, called p5live, that I think replicates what #1651 is intended for. See #1817 or read more at https://twitter.com/jgrizou/status/1371882296375132162

I think dispatch changes and propagation is somewhat linked, and p5live could help provide a simple use-case to think about.