aws s3 silently fails to handle filesystem exceptions (e.g. broken symlinks)

[alexanderAtGEn]

Symptoms

When trying to sync/cp from a local directory to S3, aws cli will exit silently, and with a non-error (0) exit status, if it runs into a filesystem exception. This issue can be reproduced with aws-cli v1.2.0.

I noticed this problem while trying to sync a directory hierarchy that contained a broken symlink. The sync would begin, and then prematurely stop before all files had been sync'ed up, with no clear explanation as to why.

Turning on debug will provide the actual exception. In my case, it was an os.stat on a broken symlink:

2013-10-21 20:45:01,071 - awscli.customizations.s3.s3handler - DEBUG - Exception caught during task execution: [Errno 2] No such file or directory: '/tmp/aws-s3-testdir/anotherdir/brokensymlink'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/s3handler.py", line 76, in call
    total_files, total_parts = self._enqueue_tasks(files)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/s3handler.py", line 150, in _enqueue_tasks
    for filename in files:
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/comparator.py", line 80, in call
    src_file = advance_iterator(src_files)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/filegenerator.py", line 49, in call
    for src_path, size, last_update in file_list:
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/filegenerator.py", line 87, in list_files
    for x in self.list_files(file_path, dir_op):
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/filegenerator.py", line 90, in list_files
    size, last_update = get_file_stat(file_path)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/utils.py", line 77, in get_file_stat
    stats = os.stat(path)
OSError: [Errno 2] No such file or directory: '/tmp/aws-s3-testdir/anotherdir/brokensymlink'

---

Reproducing The Problem

Create a set of test subdirectories/files, along with a broken symlink.

$ mkdir test-sync-dir
$ cd test-sync-dir/
$ mkdir somedir
$ cd somedir/
$ touch textfile.txt
$ cd ..
$ mkdir anotherdir
$ cd anotherdir/
$ ln -s nowhere brokensymlink
$ cd ../..

Setup an S3 bucket with nothing in it, to sync into.

$ aws s3 mb s3://my-test-bucket
$ aws s3 ls s3://my-test-bucket/

Bucket: my-test-bucket
Prefix: 

      LastWriteTime     Length Name
      -------------     ------ ----

Attempt to sync your test subdirectory into S3.

$ aws s3 sync test-sync-dir s3://my-test-bucket/
$ echo $?
0
$ aws s3 ls s3://my-test-bucket/

Bucket: my-test-bucket
Prefix: 

      LastWriteTime     Length Name
      -------------     ------ ----

Note that nothing was put into the bucket, and no errors were reported.

Now, remove the broken symlink.

$ rm anotherdir/brokensymlink 
$ aws s3 sync test-sync-dir s3://my-test-bucket/
$ echo $?
0
$ aws s3 ls s3://my-test-bucket/

Bucket: my-test-bucket
Prefix: 

      LastWriteTime     Length Name
      -------------     ------ ----
                           PRE somedir/
$ aws s3 ls s3://my-test-bucket/somedir/

Bucket: my-test-bucket
Prefix: somedir/

      LastWriteTime     Length Name
      -------------     ------ ----
2013-10-21 20:13:39          0 textfile.txt

Note that files were sync'ed to the S3 bucket.

[jamesls]

Looks like there's two things needed to fix this:

1. Automatically skip over broken symlinks
2. Make sure we have a non zero RC if we encounter any bad symlinks to let the user know that not all their files were synced.

[alexanderAtGEn]

@jamesis for number 2 I'd say: return non-success if there's any indication that files that were requested to be put on S3 didn't make it there. For my uses, I'd be happy if the exception that happens is propagated up to the user in some form and the sync/cp stops dead in its tracks.

[jamesls]

So perhaps (1) should be something you opt into, e.g --skip-bad-symlinks or something, and the default behavior is to propagate an error message and a non zero RC once we hit a bad symlink.

[xueshanf]

I have a similar problem with a mounted file system. For example,

aws s3 sync $src $dest --exclude ".backup/" --exclude ".git/" --exclude "
.svn/*" --delete

Completed 12 part(s) with ... file(s) remaining
[Errno 19] No such device: './images/.backup/.backup'

The $src is in an AFS file system (I suspect it would have similar problem with NFS file system).

The last component .backup is a AFS mounted mount point, which doesn't exist anymore.

The entire images directory would be skipped, no other files get copied over. --exclude doesn't seem to work for a directory that has file system level problem underneath.

Maybe to provide an option to skip and report errors but copy other objects over?

Xueshan