S3BotoStorage fails when keys contain ":"

[joharohl]

Trying to do this:
storage.delete("2017-01-13T12:55:32.497821+00:00/somekey")
Which fails like so:

  File "...", line 76, in ...
    storage.delete("2017-01-13T12:55:32.497821+00:00/somekey")
  File ".../local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 438, in delete
    name = self._normalize_name(self._clean_name(name))
  File "/.../local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 364, in _normalize_name
    name)
django.core.exceptions.SuspiciousOperation: Attempted access to "2017-01-13T12:55:32.497821+00:00/somekey" denied.

I have tracked this down to storages.backends.s3boto.safe_join's call to urlparse.urljoin. Having ":" confuses the join to skip the / at the beginning of final_path which then fails during the check that final_path is a sub path of base.

It's probably a small fix, but I unfortunately don't have enough knowledge of the urlparse lib to know how to do it.

PS.
And yes, AWS supports having ":" part of the keys. Running aws s3 rm s3://<bucket>/2017-01-13T12:55:32.497821+00:00/somekey works fine.

[jschneier]

Indeed. According to Amazon's Docs the colon (":") should probably be url-encoded. That is a bug in the library, probably we should make safe_join significantly smarter and ensure it complies with the docs.

[jschneier]

Out of curiosity how are you getting the file up to the server? If I use the normal Django fileuploader process then those characters are stripped out.

[jschneier]

Also, renaming a file to have : as you do via the aws cli (which I believe uses boto3 anyway) urlencodes the characters.

[joharohl]

This isn't so much of an issue in for the user uploads if it strips annoying characters then.

Our use case is for a backup script that uses datetimes for the folder names to be able to prune backups later solely based on the directory names. So we are mostly using django-storages as a convenient way to reuse the storage definitions we already use. We could of course just use boto3 directly, but it would be nice if it worked directly with django-storages.

So don't feel like it's too much of a priority to fix.

[jschneier]

Ah, makes sense. What does the actual url that amazon shows you look like in S3? My hunch is it is already urlencoded which would make this a pretty easy fix.

[joharohl]

Hi, sorry for late reply. Yes, the colons in the url are urlencoded.

So 2017-01-13T12:55:32.497821+00:00 turns into 2017-01-13T12%3A55%3A32.497821%2B00%3A00