Add directory tracking to sync

[pietern]

Changes

This change replaces usage of the repofiles package with the filer package to consolidate WSFS code paths.

The repofiles package implemented the following behavior. If a file at foo/bar.txt was created and removed, the directory foo was kept around because we do not perform directory tracking. If subsequently, a file at foo was created, it resulted in an fs.ErrExist because it is impossible to overwrite a directory. It would then perform a recursive delete of the path if this happened and retry the file write.

To make this use case work without resorting to a recursive delete on conflict, we need to implement directory tracking as part of sync. The approach in this commit is as follows:

1. Maintain set of directories needed for current set of files. Compare to previous set of files. This results in mkdir of added directories and rmdir of removed directories.
2. Creation of new directories should happen prior to writing files. Otherwise, many file writes may race to create the same parent directories, resulting in additional API calls. Removal of existing directories should happen after removing files.
3. Making new directories can be deduped across common prefixes where only the longest prefix is created recursively.
4. Removing existing directories must happen sequentially, starting with the longest prefix.
5. Removal of directories is a best effort. It fails only if the directory is not empty, and if this happens we know something placed a file or directory manually, outside of sync.

Tests

• Existing integration tests pass (modified where it used to assert directories weren't cleaned up)
• New integration test to confirm the inability to remove a directory doesn't fail the sync run

[shreyas-goenka]

Looks good to me, just needs integration tests for two cases:

1. We delete empty directory trees on the workspace
2. We do not delete if the directory tree is not empty (ie has a file in it)

[fjakobs]

Let's try this out

[shreyas-goenka]

Looks good, just some minor suggestions

[pietern]

@shreyas-goenka Could you take a look at the dirset*.go files?

[shreyas-goenka]

LGTM