"Upload files" button sometimes fails to upload #1014

stefanfoulis · 2017-10-26T13:54:55Z

The "Upload Files" button on a directory sometimes does not work. When it fails it will appear to upload the whole file, show the "success" message, but the file will be nowhere to be found.

This has been happening with "larger" pdf files around 3MB. Uploading the same file through drag'n'drop works.

tdsymonds · 2017-11-08T14:14:42Z

I'm having the same issue but with all uploads via the button, irrelevant of size. I am storing the images remotely in S3 though, I'm not sure if that has any impact. Dragging and dropping as you mention works fine for me. I've added the error below if that helps.

Internal Server Error: /admin/filer/clipboard/operations/upload/no_folder/

ParamValidationError at /admin/filer/clipboard/operations/upload/no_folder/
Parameter validation failed:
Invalid type for parameter ContentType, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

Request Method: POST
Request URL: /admin/filer/clipboard/operations/upload/no_folder/?qqfile=my_image.jpg
Django Version: 1.11.7
Python Version: 3.5.2

Note: the file I uploaded was 13KB.

Thanks

xela7 · 2017-12-14T01:14:00Z

I am also getting this same issue.

I am also using S3. Drag and drop works. Upload Button doesn't.

Traceback (most recent call last):
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/ubuntu/.virtualenvs/dev/src/django-filer/filer/admin/clipboardadmin.py", line 114, in ajax_upload
    file_obj.save()
  File "/home/ubuntu/.virtualenvs/dev/src/django-filer/filer/models/imagemodels.py", line 53, in save
    super(Image, self).save(*args, **kwargs)
  File "/home/ubuntu/.virtualenvs/dev/src/django-filer/filer/models/abstract.py", line 74, in save
    super(BaseImage, self).save(*args, **kwargs)
  File "/home/ubuntu/.virtualenvs/dev/src/django-filer/filer/models/filemodels.py", line 194, in save
    super(File, self).save(*args, **kwargs)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/polymorphic/models.py", line 74, in save
    return super(PolymorphicModel, self).save(*args, **kwargs)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/base.py", line 837, in save_base
    self._save_parents(cls, using, update_fields)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/base.py", line 864, in _save_parents
    self._save_table(cls=parent, using=using, update_fields=update_fields)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/base.py", line 924, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/base.py", line 963, in _do_insert
    using=using, raw=raw)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/query.py", line 1076, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1106, in execute_sql
    for sql, params in self.as_sql():
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1059, in as_sql
    for obj in self.query.objs
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1008, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 296, in pre_save
    file.save(file.name, file.file, save=False)
  File "/home/ubuntu/.virtualenvs/dev/src/django-filer/filer/fields/multistorage_file.py", line 122, in save
    super(MultiStorageFieldFile, self).save(name, content, save)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/easy_thumbnails/files.py", line 670, in save
    super(ThumbnailerFieldFile, self).save(name, content, *args, **kwargs)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 94, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 404, in _save
    self._save_content(key, content, headers=headers)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 415, in _save_content
    rewind=True, **kwargs)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/boto/s3/key.py", line 1305, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/boto/s3/key.py", line 762, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/boto/s3/key.py", line 963, in _send_file_internal
    query_args=query_args
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/boto/s3/connection.py", line 671, in make_request
    retry_handler=retry_handler
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/boto/connection.py", line 940, in _mexe
    request.body, request.headers)
  File "/home/ubuntu/.virtualenvs/dev/local/lib/python2.7/site-packages/boto/s3/key.py", line 896, in sender
    response.status, response.reason, body)
S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>############</AWSAccessKeyId><StringToSign>PUT#######==
None
Thu, 14 Dec 2017 01:05:07 GMT
x-amz-acl:public-read
/dev.media.example.com/media/filer_public/e3/a0/e3a00fba-e612-470d-b4da-80c4d410f2ab/bass.jpg</StringToSign><SignatureProvided>#######=</SignatureProvided><StringToSignBytes>DOUBLE HEX DIGITS HERE</StringToSignBytes><RequestId>######</RequestId><HostId>#######=</HostId></Error>

Django==1.11.6
django-filer: revision #472a0419bfa185a8b0a861bd0779ac6d817082c7
python 2.7.6

bhrutledge · 2018-01-05T22:16:26Z

Same behavior as above, but I thought I'd add some (hopefully useful) color.

Python 2.7.13
Django 1.11.9
django-filer 1.3.0
boto 2.48.0

The configuration below works in Django 1.8, with older versions of boto and filer.

# settings.py
FILER_STORAGES = {
    'public': {
        'main': {
            'ENGINE': 'es.s3storages.CMSStorage',
            'UPLOAD_TO': 'filer.utils.generate_filename.randomized',
            'UPLOAD_TO_PREFIX': 'filer',
        },
        'thumbnails': {
            'ENGINE': 'es.s3storages.CMSStorage',
            'THUMBNAIL_OPTIONS': {
                'base_dir': 'thumbnails'
            },
        },
    },
}

THUMBNAIL_PROCESSORS = (
    'easy_thumbnails.processors.colorspace',
    'easy_thumbnails.processors.autocrop',
    #'easy_thumbnails.processors.scale_and_crop',
    'filer.thumbnail_processors.scale_and_crop_with_subject_location',
    'easy_thumbnails.processors.filters',
)

# s3storages.py
CMSStorage = lambda: S3BotoStorage(bucket=settings.AWS_CMS_STORAGE_BUCKET_NAME,
                                   custom_domain=settings.S3_CMS_DOMAIN,
                                   file_overwrite=False, encryption=False)

bhrutledge · 2018-01-09T17:17:26Z

A little more color: swapping out boto for boto3==1.5.10 and using S3Boto3Storageshows the same behavior ("Upload Files" fails, drag & drop works), but with a different error:

Internal Server Error: /admin/filer/clipboard/operations/upload/6/
Traceback (most recent call last):
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/filer/admin/clipboardadmin.py", line 114, in ajax_upload
    file_obj.save()
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/filer/models/imagemodels.py", line 53, in save
    super(Image, self).save(*args, **kwargs)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/filer/models/abstract.py", line 74, in save
    super(BaseImage, self).save(*args, **kwargs)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/filer/models/filemodels.py", line 194, in save
    super(File, self).save(*args, **kwargs)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/polymorphic/models.py", line 74, in save
    return super(PolymorphicModel, self).save(*args, **kwargs)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/base.py", line 837, in save_base
    self._save_parents(cls, using, update_fields)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/base.py", line 864, in _save_parents
    self._save_table(cls=parent, using=using, update_fields=update_fields)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/base.py", line 924, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/base.py", line 963, in _do_insert
    using=using, raw=raw)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/query.py", line 1076, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1111, in execute_sql
    for sql, params in self.as_sql():
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1064, in as_sql
    for obj in self.query.objs
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1013, in pre_save_val
    return field.pre_save(obj, add=True)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/fields/files.py", line 296, in pre_save
    file.save(file.name, file.file, save=False)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/filer/fields/multistorage_file.py", line 122, in save
    super(MultiStorageFieldFile, self).save(name, content, save)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/easy_thumbnails/files.py", line 670, in save
    super(ThumbnailerFieldFile, self).save(name, content, *args, **kwargs)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/db/models/fields/files.py", line 94, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 452, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 467, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/Users/brian/.virtualenvs/es-111/lib/python2.7/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
ParamValidationError: Parameter validation failed:
Invalid type for parameter ContentType, value: None, type: <type 'NoneType'>, valid types: <type 'basestring'>
"POST /admin/filer/clipboard/operations/upload/6/?qqfile=solar-panels-ground.jpg HTTP/1.1" 500 45481
"GET /admin/filer/folder/6/list/?order_by=-modified_at HTTP/1.1" 200 19905
"GET /admin/jsi18n/ HTTP/1.1" 200 3217

twoblokeswithapostie · 2018-03-26T06:57:22Z

Experiencing same issue with uploads to S3. Drag and drop works, but Upload Files doesn't. "ContentType" inside parameters dictionary is None. One of the reporters at boto/boto3#1256 is using filer and experiencing the same issue.

twoblokeswithapostie · 2018-03-26T07:04:59Z

The difference I see here is that the Content-Type by fileuploader.min.js is set to "Content-Type: application/octet-stream" when the button is used. And when the files are dropped dropzone.min.js uses "Content-Type: multipart/form-data".

bhrutledge · 2018-03-26T13:57:01Z

Some observations:

boto/boto3#1256 leads to jschneier/django-storages#407, which seems to be a workaround for a None ContentType. It's merged, but not released.

Following the URL in fileuploader.min.js led me to it's replacement, https://github.com/FineUploader/fine-uploader.

It might be informative to try either of these things out.

bhrutledge · 2018-03-26T20:47:40Z

I installed the latest master of django-storages, and am no longer seeing this issue in my local environment. However, it's unclear when we'll get a new release of django-storages.

bhrutledge · 2018-03-27T14:19:13Z

I pinged the maintainer of django-storages, and he released a new version with the Content-Type fix. I confirmed that it resolves the issue in my local environment. Haven't tried in production yet. Can any of you confirm?

xela7 · 2018-03-27T17:03:37Z

I'll take a look later today or tomorrow. Thanks for your work!

…

On Tue, Mar 27, 2018 at 7:19 AM, Brian Rutledge ***@***.***> wrote: I pinged the maintainer of django-storages, and he released a new version <https://github.com/jschneier/django-storages/blob/master/CHANGELOG.rst#166-2018-03-26> with the Content-Type fix. I confirmed that it resolves the issue in my local environment. Haven't tried in production yet. Can any of you confirm? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1014 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHr7pZhP_y_gFohmUcQ7hcdPYJEDuprks5tikpjgaJpZM4QHnD3> .

twoblokeswithapostie · 2018-04-11T00:54:50Z

I can confirm that upgrading to django-storages 1.6.6 fixes this issue. Thank you for your help!

xela7 · 2018-04-11T02:18:35Z

Oh jeez ! Thanks Mario! I lost track of this issue. - Alex

…

On Tue, Apr 10, 2018 at 5:54 PM, Mario Gudelj ***@***.***> wrote: I can confirm that upgrading to django-storages 1.6.6 fixes this issue. Thank you for your help! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1014 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHr7oNs55K1CU9bgsUnsoiD2KL9Y0u8ks5tnVRbgaJpZM4QHnD3> .

yakky · 2018-04-14T19:59:31Z

@bhrutledge thanks for your investigation!
Thanks @xela7 @twoblokeswithapostie @tdsymonds for your inputs
I'm going to close this

stefanfoulis added the bug label Oct 26, 2017

bhrutledge mentioned this issue Mar 26, 2018

Detect Content-Type when content_type property is null jschneier/django-storages#407

Merged

yakky closed this as completed Apr 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Upload files" button sometimes fails to upload #1014

"Upload files" button sometimes fails to upload #1014

stefanfoulis commented Oct 26, 2017

tdsymonds commented Nov 8, 2017 •

edited

Loading

xela7 commented Dec 14, 2017 •

edited

Loading

bhrutledge commented Jan 5, 2018 •

edited

Loading

bhrutledge commented Jan 9, 2018

twoblokeswithapostie commented Mar 26, 2018

twoblokeswithapostie commented Mar 26, 2018

bhrutledge commented Mar 26, 2018

bhrutledge commented Mar 26, 2018

bhrutledge commented Mar 27, 2018

xela7 commented Mar 27, 2018 via email

twoblokeswithapostie commented Apr 11, 2018

xela7 commented Apr 11, 2018 via email

yakky commented Apr 14, 2018

"Upload files" button sometimes fails to upload #1014

"Upload files" button sometimes fails to upload #1014

Comments

stefanfoulis commented Oct 26, 2017

tdsymonds commented Nov 8, 2017 • edited Loading

xela7 commented Dec 14, 2017 • edited Loading

bhrutledge commented Jan 5, 2018 • edited Loading

bhrutledge commented Jan 9, 2018

twoblokeswithapostie commented Mar 26, 2018

twoblokeswithapostie commented Mar 26, 2018

bhrutledge commented Mar 26, 2018

bhrutledge commented Mar 26, 2018

bhrutledge commented Mar 27, 2018

xela7 commented Mar 27, 2018 via email

twoblokeswithapostie commented Apr 11, 2018

xela7 commented Apr 11, 2018 via email

yakky commented Apr 14, 2018

tdsymonds commented Nov 8, 2017 •

edited

Loading

xela7 commented Dec 14, 2017 •

edited

Loading

bhrutledge commented Jan 5, 2018 •

edited

Loading