Apply smart_text on file names in python2.7 [s3boto3]

[g-as]

Not a 100% sure it's the best approach, but there it is: my attempt to address #216.

I feel that _encode_name and _decode_name are starting to lose their naming/meaning values since they now end up doing almost the same thing.

Also, I didn't know what to precisely test. The only thing I know for sure is that the test I added fail without the switch from smart_str to smart_text.

[codecov-io]

Codecov Report

Merging #217 into master will decrease coverage by 13.14%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           master     #217       +/-   ##
===========================================
- Coverage   72.48%   59.33%   -13.15%     
===========================================
  Files          10       17        +7     
  Lines        1381     1692      +311     
===========================================
+ Hits         1001     1004        +3     
- Misses        380      688      +308

Impacted Files	Coverage Δ
storages/backends/s3boto3.py	81.92% <100%> (-3.03%)	⬇️
storages/backends/s3boto.py	86.13% <0%> (-0.23%)	⬇️
storages/backends/hashpath.py	85.71% <0%> (ø)
storages/backends/image.py	0% <0%> (ø)
storages/backends/couchdb.py	0% <0%> (ø)
storages/backends/overwrite.py	0% <0%> (ø)
storages/backends/database.py	0% <0%> (ø)
storages/backends/symlinkorcopy.py	0% <0%> (ø)
storages/backends/mogile.py	0% <0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ad4890...60d2d51. Read the comment docs.

[belak]

This also fails in the s3boto backend.

[jschneier]

Okay, let me add a test and then I'll merge...

[jschneier]

Oh you have a test! How awesome.

[jschneier]

@belak thanks for pointing that out. As soon as I merge this I will fix it in the Boto backend as well.

[jschneier]

I'm happy to merge this but I can't quite get the UnicodeDecodeError to manifest even if I remove the fix and keep the tests. Any pointers on how to get that to happen?

[g-as]

Hey,

rolling back my proposed fixed but keeping the added/modified tests, I get this for each python2.7 test suite:

self = <tests.test_s3boto3.S3Boto3StorageTests testMethod=test_special_characters>

    def test_special_characters(self):
        self.storage.custom_domain = "mock.cloudfront.net"

        name = "ãlöhâ.jpg"
        content = ContentFile('new content')
        self.storage.save(name, content)
>       self.storage.bucket.Object.assert_called_once_with(name)

tests/test_s3boto3.py:330:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.tox/py27-django18/lib/python2.7/site-packages/mock.py:846: in assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_mock_self = <MagicMock name='mock.Bucket().Object' id='139854456779664'>, args = ('ãlöhâ.jpg',), kwargs = {}, self = <MagicMock name='mock.Bucket().Object' id='139854456779664'>
msg = "Expected call: Object(u'\\xe3l\\xf6h\\xe2.jpg')\nActual call: Object('\\xc3\\xa3l\\xc3\\xb6h\\xc3\\xa2.jpg')"

    def assert_called_with(_mock_self, *args, **kwargs):
        """assert that the mock was called with the specified arguments.

            Raises an AssertionError if the args and keyword args passed in are
            different to the last call to the mock."""
        self = _mock_self
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            raise AssertionError('Expected call: %s\nNot called' % (expected,))

        if self.call_args != (args, kwargs):
            msg = self._format_mock_failure_message(args, kwargs)
>           raise AssertionError(msg)
E           AssertionError: Expected call: Object(u'\xe3l\xf6h\xe2.jpg')
E           Actual call: Object('\xc3\xa3l\xc3\xb6h\xc3\xa2.jpg')

.tox/py27-django18/lib/python2.7/site-packages/mock.py:835: AssertionError
------------------------------------------------------------------------------------------ Captured stderr call -------------------------------------------------------------------------------------------
/home/vagrant/code/django-storages-wam/.tox/py27-django18/lib/python2.7/site-packages/mock.py:2075: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  return (other_args, other_kwargs) == (self_args, self_kwargs)
_______________________________________________________________________________ S3Boto3StorageTests.test_storage_open_write _______________________________________________________________________________

self = <tests.test_s3boto3.S3Boto3StorageTests testMethod=test_storage_open_write>

    def test_storage_open_write(self):
        """
            Test opening a file in write mode
            """
        name = 'test_open_for_writïng.txt'
        content = 'new content'

        # Set the encryption flag used for multipart uploads
        self.storage.encryption = True
        self.storage.reduced_redundancy = True
        self.storage.default_acl = 'public-read'

        file = self.storage.open(name, 'w')
>       self.storage.bucket.Object.assert_called_with(name)

tests/test_s3boto3.py:176:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_mock_self = <MagicMock name='mock.Bucket().Object' id='139854456294352'>, args = ('test_open_for_writïng.txt',), kwargs = {}, self = <MagicMock name='mock.Bucket().Object' id='139854456294352'>
msg = "Expected call: Object(u'test_open_for_writ\\xefng.txt')\nActual call: Object('test_open_for_writ\\xc3\\xafng.txt')"

    def assert_called_with(_mock_self, *args, **kwargs):
        """assert that the mock was called with the specified arguments.

            Raises an AssertionError if the args and keyword args passed in are
            different to the last call to the mock."""
        self = _mock_self
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            raise AssertionError('Expected call: %s\nNot called' % (expected,))

        if self.call_args != (args, kwargs):
            msg = self._format_mock_failure_message(args, kwargs)
>           raise AssertionError(msg)
E           AssertionError: Expected call: Object(u'test_open_for_writ\xefng.txt')
E           Actual call: Object('test_open_for_writ\xc3\xafng.txt')

.tox/py27-django18/lib/python2.7/site-packages/mock.py:835: AssertionError

Here is the full tox output:
storages_test_report.txt

[jschneier]

@AGASS007 yes I did just that. But I still don't see the UnicodeDecodeError that you mentioned in #216. I understand what the issue between the test diffing (code point/encoded bytes) but would like to see the exception!

[g-as]

@jschneier I pasted a stacktrace in the issue #216

[nschlemm]

@AGASS007 thanks for fixing this!
@jschneier please integrate this into the next release. THX :)

[jschneier]

Okay. Interestingly I cannot reproduce this on the non-Boto3 backend. I'm going to then remove _decode_name. This looks good.

[jschneier]

@belak you mentioned this failing for you on the boto backend, i could not reproduce. have a failing test case?

[belak]

I don't unfortunately. If I manage to reproduce it, I'll open an issue.