Sync only the file change, not entire file [$1,755]

[wesleyhuang]

In Dropbox, there notes that only the file changes are sync-ed, not the entire file. https://www.dropbox.com/help/8/en. It is great if ownCloud can do the same. Especially for large files, sync-ing the entire files make a lot of bandwidth and time unnessararily wasted.

In my testing with latest 4.5.4 and sync client in Ubuntu 12.04, I prepared a 1GB text file, append a few characters to the end, and monitor the traffic and the file in the server, I see the entire 1GB file is transffered to the server and the server is actually creating a new file.

A reply from the forum indicates that librsync has this feature http://librsync.sourceforge.net/, maybe the csync can be switched to the libsync.

[allo-]

i think this is ESSENTIAL for a sync client. And detecting of moved files. Maybe combining this like rsync -y (trying to find a similiar base file on the remote side to speedup upload), but note that rsync -y can slow down (lib)rsync when working in large directories.

[rarspace01]

any update on this?

[LoZio]

+1 for this. There's a wealth of files that may slightly change each time and are big to sync. Outlook PST, truecrypt, databases, phone backups, ...
Any update on this?

[jmstacey]

I'd like to see this as well.

[Dont-ask-for-it]

Waiting on this to install my first owncloud server.

[notDavid]

I hope this will be implemented, seems essential..

• for some reason my ownCloud client (1.3.0, OS X) has trouble syncing big files with small changes, and always re-uploads entire files...

[tityrus]

I also see this as essential. Please.

[xeloader]

I agree on this one as well, we need to be able to have some kind of incremental sync possibility before this is 100% usable.

[dragotin]

The problem is known, and we will get to it. At one day.

Until then, please try to retain from +1 comments ✌️

[Poelziminator]

Can you give an estimate when will see the “one day”?
This functionality would open up whole new possibilities like syncing Truecrypt-volume files or Lightroom catalogs.

[danimo]

@Poelziminator Not in any release this year. Note that most work for this will be in the server and the general design.

[zwo-bot]

This is the key-feature for everone works with trucrypt or bigger dbfiles. Please make it happen.

[jancborchardt]

This as well seems related to the 1.6 »Sync Performance« milestone. @MTRichards @dragotin @danimo?

[MTRichards]

This is delta file syncing, and likely too complicated to get into 1.6 because it requires major server work too. The idea is to first improve performance on file level sync to get it more efficient, and then increase the granularity of the file comparisons (to file chunks), but going right to file chunks without first getting the file level sync comparisons would hurt performance more than help at this point because of the sheer volume of comparisons required.

[curtisz]

I was about to choose ownCloud (with a very high probability of buying Enterprise) for use at our company but decided against it because of this particular issue. I'm a bit shocked that this isn't supported, considering how important this is for (potential) customers using TrueCrypt et al. This isn't so much of a +1 as an "at least one enterprise customer lost to the other guys".

[huksley]

This is very important for virtual machine disk image files. Single byte change causes upload of multi-gigabyte files to remote server.

[beniroquai]

Yes! Great idea!

[powerpaul17]

As this is really the most important feature of a cloud service and it seems that nobody is interested in or working on it, I would like to offer some help with this issue. Is there already some information about what needs to be done, where to start, etc.?

[sagar-srivastava]

I agree this is a very important feature. Owncloud is such a wonderful piece of software. I tested it today and found its quality up to the mark. Just Delta file sync addition will make it complete. Web interface/WebDav/Desktopsync/file sharing/ has worked out great on my VPS, works with ISPConfig3 implementation.

please initiate this effort and I am willing to buy the enterprise edition for my company.

[sodetemplin]

This point should be an ownCloud priority, for sure ! Without this functionnality, the EE version is "just" the community one with more support ?

[kevincox]

I remember hearing that owncloud wanted to keep the files stored on disk in their entirety. Is this (still) true? Because if so you could just generate a zsync signature file and a custom receiver that generated the entire file.

If the files are allowed to be broken up (maybe in a future version) then they can be chunked and a very efficient sync endpoint could be made.

What are the thoughts on this. I may consider working on this is my spare time.

[jancborchardt]

@dragotin @danimo @MTRichards what’s the plan on this one? I know it’s a big one, but it’s requested very often and seems to be important to improve performance.

[danimo]

@jancborchardt As long as the server doesn't offer any delta syncs, we can't implement it.

[L0j1k]

@danimo Can you explain what you mean in a little more detail, please?

[danimo]

@LoZio Currently, we use WebDAV as the communication protocol with the server. Additionally, we can upload in chunks, but only if we transfer the entire file. Delta-sync requires another protocol extension. Also, we have no gurantee that the server is holding the hash-wise same file, since the server does not store file hashes.

[jancborchardt]

So what’s the plan with the server-side regarding this one? @karlitschek @DeepDiver1975 @PVince81

[karlitschek]

We should do this in the future. But it is more a long term feature

[DeepDiver1975]

Technically zsync could detect moves, but my code doesn’t support that, all bytes moved will be resent, this was to make it easier to get something working quickly. I mentioned this as a future work area.

in which area are the adoptions neccessary to make this work? I'm trying to find out the impact - if bigger refactorings are necessary I question the current in depth review ...

[PVince81]

does the current impl at least help if appending stuff at the end of files ? In this case I expect that since the beginning's offset did not change it would only sync the appended block.

[PVince81]

did anyone here already test the feature ? if you did, please report where you saw improvements (file types, use case, etc)

[ahmedammar]

Appending will only send appended bytes. Regarding where the work needs to be done:

1. zsync - code wasn’t designed with upload path in mind, I implemented that and use the path of least resistance, meaning dropping moved block support.
2. oC - again to support moved blocks we’d need a more complex upload path where moved chunks are processed somehow, this can be as simple as sending a file with moved block to:from and modifying assembly code to handle appropriately.

I recommend we don’t try to do this now but after current simple approach is well tested.

[ahmedammar]

Just wanted to clarify something, the above only applies to the uploader, the downloader will not redownload moved chunks.

[gpothier]

@PVince81 Regarding the file types that would benefit from delta sync: we are a publishing house and we work with rather big (up to 500MB) Adobe InDesign files. Small changes to these files are very good candidates for delta sync (Dropbox syncs small changes to these files almost instantly).

[ckamm]

The feature is stabilizing in the delta-sync branch and test builds will become available around the time 2.5.0 is released. If things go smoothly it'll be in 2.6.0.

[KevinLeigh]

Is this issue currently being resolved? Would like to start resolving this issue but do not want to if someone else is.

[ahmedammar]

Yeah would be nice if this gets closed so I can claim the bounty?

[ahmedammar]

Is this really still not merged? What's the hold-up guys? Can we close this ticket??

[guruz]

@ahmedammar It will be in client 2.6.0
On Friday we released 2.5.0 beta1. So it's coming :)
I don't know how the bounty thing works, but i'll try to get that info.

[DeepDiver1975]

@guru was the PR finally merged? Yes? Close Ticket and we Trigger payment on Bountysource

[ahmedammar]

Thanks! Have submitted the bounty claim too ...

[curtiszimmerman]

@ahmedammar I don't use Owncloud anymore, but I was one of the contributors to the Bountysource bounty, and I just wanted to thank you for all this work. It was very interesting (and impressive) to read your status updates and commits. I hope you buy at least one beer from the bounty (or a nice dinner)!

[hodyroff]

Thanks @ahmedammar really appreciated your contribution and looking forward to your continued participation - as time fits. Delta sync will for sure help to bring ownCloud into new places and motivate people to contribute and use it! Privacy and security and a full concentration on file sync and share are so important in todays world. Again Thank You!

[trampi]

I know that one should not post off-topic comments in issues. However, I want to thank @ahmedammar and all other contributors in this issue. This is a really nice and long awaited feature. Well done, everyone!

[LoZio]

I also want to thank you @ahmedammar for you work. My first post above is from 5 years ago and I'll be happy to go back to OC if this works!

[warnerbryce]

@LoZio it’s gonna be merged to Nextcloud too.

[ahmedammar]

Hi all, thanks for all the kind words, much appreciated, it was a fun project and I had fun doing it, hope it stands the test of time. Would like to extend the thanks to all the oC devs who helped throughout the process and especially @ckamm who has so aptly continued the development where I left off, very nice! Thanks again all!

[guruz]

This feature will be in the upcoming 2.6 alpha release.
Meanwhile you can try it inside the daily builds 2.6.x https://download.owncloud.com/desktop/daily/

[bolandross]

So, let's say I wanted to experiment with this new feature. I just downloaded the 2.6.0alpha1 Desktop Client and enabled delta synchronisation. Unfortunately, without noticing any effect. So my question is: Which ownCloud server version is necessary for this to work as intended?

Thanks in advance and keep up the good work.

[michaelstingl]

@bolandross The server part is only in the master branch of the ownCloud server. I hasn't been released in a production version yet.

[chrisgraham]

Somebody needs to update the FAQ: https://owncloud.org/faq/#partialsyncing