Using file.write() fails on S3 when files are missing

[vdboor]

This works:

storage = S3Boto3Storage()
storage.file_overwrite = True
storage.save('test.txt', BytesIO(b'data'))

This fails:

file = storage.open('test.txt', 'w')
file.write(b'data')
file.close()

The S3Boto3StorageFile.write() method tries to detect the file size (self.file.tell()) which raises an exception when the file doesn't exist yet.

>>> f = storage.open('test.txt', 'w')
>>> f.write(b'test')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 128, in write
    if self.buffer_size <= self._buffer_file_size:
  File "/usr/local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 134, in _buffer_file_size
    pos = self.file.tell()
  File "/usr/local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 99, in _get_file
    if self._storage.gzip and self.obj.content_encoding == 'gzip':
  File "/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 339, in property_loader
    self.load()
  File "/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 505, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 623, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

[holmberd]

Related, #350

[divick]

No activity on this since more than a year. This started to happen for me suddenly and i am wondering what really changed for this to fail. I am still wondering if it is really a django-storages issue?

[michaelfward]

I'm experiencing the issue as well. Any workarounds?

[michaelfward]

Seems that open(), close(), and then open() + write() is a workaround for now

[nl-emmanuellodovice]

I had this problem also, my work around was to add a save_overwrite method

    def save_overwrite(self, name, content, max_length=None):
        """ Save method that makes sure the file is saved in the provided name by deleting the existing file if there is
         one. This method is useful when you are storing a file using a computed path and the path is not stored in
         the database.
        """
        if self.exists(name):
            self.delete(name)
        return self.save(name, content, max_length)

[jschneier]

Okay will fix, the issue here is that the gzip downloading path should only be called on files opened in read mode, not write.