Improve performance for large folders

[greezybacon]

First of all, thank you for the project. I think it's very promising and very elegantly implemented.

I'll offer my use case first, as it might differ from others' usage of the project. I have two servers, we'll call them A and B. They are network file servers and should accept writes from any client at any time. It is my goal to use bitpocket to synchronize the writes between the two boxes. And yes, I'm aware that a cluster file system with distributed locking might be a better option; however, I think that's overly complicated. In our environment, folks don't generally edit the same file at the same time. Having a 10 minute lag between the servers is acceptable. I'm using the idea to replace Microsoft(R) DFSR (Distributed File System Replication), which works similarly to this project. (with the exception of a GUI)

When I started testing bitpocket I noticed that it deleted files unexpectedly and that it also pushed deleted files back. I believe this is because of how it tracks added and deleted files. Although describing why I think that is in a few paragraphs is very difficult.

I've adjusted how bitpocket works so that it assumes that, after the sync is complete, both of the servers have the same contents. Then, when the sync is performed the next time, a file list from both servers is collected and compared against the previous sync listing. A list of locally added and deleted files and remotely deleted files is compiled. These two lists are used for the pull and push to ensure that none of the files is deleted nor resurrected unexpectedly.

The most significant change here is that the stock version did not use any exclude list for the push.

I've also adjusted the backup method to run in parallel with the pull sync. This reduces one of the rsyncs between the servers. In my environment, I have about 500G of data in about 250k files. So each rsync between the servers is time consuming. And what's more, there is a bug in recent versions of rsync such that, when extended attributes are synced (--xattrs), then the sync time goes from about a minute to about 20 minutes.

I can go into more detail for any of the edits or commits but mainly want to start the conversation. I also don't mind rebasing to change parts of it.

Cheers,

[standin000]

can't reproduce disappearing and reappearing files scenario in master version, How do you meet it? @greezybacon

[greezybacon]

I've also attempted to figure it out. I worked at trying to piece back together what I noticed when bitpocket had troubles. I tried putting together a test to reproduce it. I put this together, which the master branch fails. I think the problems surround the usage of the added-prev file, since it adds exclusions from the following sync. Strangely, my branch fails this test too.

diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb
index 3cc3e83..1142014 100644
--- a/spec/spec_helper.rb
+++ b/spec/spec_helper.rb
@@ -35,6 +35,10 @@ def rm(path)
   FileUtils.rm(path)
 end
 
+def mv(a, b)
+  FileUtils.mv(a, b)
+end
+
 RSpec::Matchers.define :exist do
   match do |filename|
     File.exist?(filename)
diff --git a/spec/sync_spec.rb b/spec/sync_spec.rb
index 080c73e..a7953c9 100644
--- a/spec/sync_spec.rb
+++ b/spec/sync_spec.rb
@@ -75,4 +75,27 @@ describe 'bitpocket sync' do
     local_path('a').should_not exist
     remote_path('a').should_not exist
   end
+
+  it 'handles remote deletes between syncs' do
+    touch remote_path('a/c')
+    touch remote_path('a/f')
+
+    sync.should succeed
+
+    # After the sync, 'c' and 'f' are in 'added-prev', so they are excluded
+    # from the next pull
+
+    rm remote_path('a/c')
+    mkdir remote_path('a/b')
+    mv remote_path('a/f'), remote_path('a/b/f')
+
+    sync.should succeed
+
+    local_path('a/c').should_not exist
+    local_path('a/f').should_not exist
+    local_path('a/b/f').should exist
+
+    remote_path('a/c').should_not exist
+    remote_path('a/f').should_not exist
+  end
 end

I also continue to observe that bitpocket has this side effect where some files seem to get pulled back even if they were locally edited. And sometimes the same thing happes to permissions. If they are changed locally, the old permissions will be pulled. I haven't found a solution to either of these yet.

[greezybacon]

Okay, I was able to get the test to pass on my branch.

[greezybacon]

Okay, I think most of my problems may have been self-inflicted. Since I was attempting to work around the performance issues, I probably introduced the strange behavior. I was able to fix the syncing of permissions.

[aggsol]

Will this ever be merged?

[greezybacon]

@aggsol I don't have the authority to merge this. But I am using bitpocket with these patches for mission-critical data syncing. I don't know if @sickill has any interest in continuing to maintain the product. So perhaps I should apply to become a maintainer...

[ku1ik]

Hey @greezybacon, I'm happy to invite you as one of the maintainers of this project 🎉

I don't really have time and motivation to keep this project going, so I hope you and @torfason (if he's still interested / using it) can keep it in good health.

[torfason]

Hi Jared (@greezybacon) and Marcin (@sickill),

I have not reviewed in detail each the commits that Jared has been submitting, but I read through a number of them. They are thoughtful and in line with what I saw as the key attraction of bitpocket - it being a simple to set up batch synchronizer with as minimal dependencies as possible, and as robust as possible.

It seems to me that Jared is currently the most dedicated user and developer of bitpocket, and I would vote to make him maintainer. It would be great to see this product continue to be improved.

Best,
Magnus

ps. I don't see myself contributing significantly to the project in the near future, so if Jared takes on the mantle, it seems it would be in the role of lead maintainer, which would be great from my perspective.

[ku1ik]

Go for it @greezybacon! ;)

[greezybacon]

The issues that two of the three last commits fixed exist in the master branch, I believe. I'll merge this branch and continue to track any issues I notice as new issues. I'll also go through the existing issues and pull requests to see which ones still apply or need to be rebased.

[aggsol]

If you feel up to the task, removing the "maintainers needed" notice from the readme won't keep new users away anymore ;-)

[ku1ik]

@aggsol thanks for pointing this out. I've just removed this notice from README.