Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance issues with WebDav (remote.php) #7964

Closed
Flying--Dutchman opened this issue Jan 19, 2018 · 4 comments
Closed

Performance issues with WebDav (remote.php) #7964

Flying--Dutchman opened this issue Jan 19, 2018 · 4 comments

Comments

@Flying--Dutchman
Copy link

Flying--Dutchman commented Jan 19, 2018 •
edited
Loading

Steps to reproduce

  1. Add WebDav network drive using nextcloud remote.php

Big file:

  1. copy a large file to nextcloud using this drive

Small files:

  1. Move a large amount from one folder to another

Expected behaviour

Big file:
Upload of the file shouldn't hang at the last 1% of the upload.

Small files:
Moving and uploading should be quick and not with a pause between every file

Actual behaviour

Big file:
Upload hangs at 99% for a couple of minutes, then it finishes or produces an error "Element not found".
First 99% upload speed is at about 200mb/s and done in about 10 seconds.

Small files:
Upload and moving takes ages.

Server configuration

Operating system:
Debian 8.10 as Host, Alpine Linux Docker image with nextcloud.
MySql is hosted by Host.

Web server:
Nginx 1.12.2-rl

Database:
MySQL 5.5.58

PHP version:
7.1.9

Nextcloud version: (see Nextcloud admin page)
12.0.4

Updated from an older Nextcloud/ownCloud or fresh install:
Fresh install

Where did you install Nextcloud from:
Docker image from linuxserver.io

Signing status:

Signing status 
No errors have been found.

List of activated apps:

App list 
Enabled:
  - activity: 2.5.2
  - audioplayer: 2.2.2
  - bookmarks: 0.10.1
  - bruteforcesettings: 1.0.3
  - calendar: 1.5.7
  - comments: 1.2.0
  - contacts: 2.0.1
  - dav: 1.3.0
  - drawio: 0.8.8
  - external: 2.0.3
  - federatedfilesharing: 1.2.0
  - federation: 1.2.0
  - files: 1.7.2
  - files_pdfviewer: 1.1.1
  - files_sharing: 1.4.0
  - files_texteditor: 2.4.1
  - files_trashbin: 1.2.0
  - files_versions: 1.5.0
  - files_videoplayer: 1.1.0
  - firstrunwizard: 2.1
  - gallery: 17.0.0
  - groupfolders: 1.1.0
  - keeweb: 0.4.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.0.0
  - nextcloud_announcements: 1.1
  - notifications: 2.0.0
  - oauth2: 1.0.5
  - password_policy: 1.2.2
  - provisioning_api: 1.2.0
  - quota_warning: 1.1.1
  - ransomware_protection: 1.0.5
  - serverinfo: 1.2.0
  - sharebymail: 1.2.0
  - survey_client: 1.0.0
  - systemtags: 1.2.0
  - tasks: 0.9.5
  - theming: 1.3.0
  - twofactor_backupcodes: 1.1.1
  - updatenotification: 1.2.0
  - workflowengine: 1.2.0
Disabled:
  - admin_audit
  - encryption
  - files_external
  - ownbackup
  - user_external
  - user_ldap

Nextcloud configuration:

Config report 
{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "datadirectory": "\/data",
        "instanceid": "oc3e67um6t92",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
			"***REMOVED SENSITIVE VALUE***"
        ],
        "overwrite.cli.url": "https:\/\/"***REMOVED SENSITIVE VALUE***"",
        "dbtype": "mysql",
        "version": "12.0.4.3",
        "dbname": "nextcloud",
        "dbhost": "192.168.178.100:3306",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "auth.bruteforce.protection.enabled": false
    },
    "apps": {
        "activity": {
            "enabled": "yes",
            "installed_version": "2.5.2",
            "types": "filesystem"
        },
        "audioplayer": {
            "enabled": "yes",
            "installed_version": "2.2.2",
            "types": "filesystem"
        },
        "backgroundjob": {
            "lastjob": "56"
        },
        "bookmarks": {
            "enabled": "yes",
            "installed_version": "0.10.1",
            "types": ""
        },
        "bruteforcesettings": {
            "enabled": "yes",
            "installed_version": "1.0.3",
            "types": ""
        },
        "calendar": {
            "enabled": "yes",
            "installed_version": "1.5.7",
            "types": ""
        },
        "comments": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "logging"
        },
        "contacts": {
            "enabled": "yes",
            "installed_version": "2.0.1",
            "types": ""
},
        "core": {
            "backgroundjobs_mode": "cron",
            "installed.bundles": "[\"CoreBundle\"]",
            "installedat": "1515456733.5354",
            "lastcron": "1516376717",
            "lastupdateResult": "[]",
            "lastupdatedat": "1516376551",
            "oc.integritycheck.checker": "[]",
            "public_files": "files_sharing\/public.php",
            "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
            "scss.variables": "d90e2783d0e072805f2aa28dc1097658",
            "vendor": "nextcloud"
        },
        "dav": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "types": "filesystem"
        },
        "drawio": {
            "enabled": "yes",
            "installed_version": "0.8.8",
            "types": "filesystem"
        },
        "external": {
            "enabled": "yes",
            "installed_version": "2.0.3",
            "types": ""
        },
        "federatedfilesharing": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": ""
        },
        "federation": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "authentication"
        },
        "files": {
            "cronjob_scan_files": "500",
            "enabled": "yes",
            "installed_version": "1.7.2",
            "types": "filesystem"
        },
        "files_pdfviewer": {
            "enabled": "yes",
            "installed_version": "1.1.1",
            "ocsid": "166049",
            "types": ""
        },
        "files_sharing": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "filesystem"
        },
        "files_texteditor": {
            "enabled": "yes",
            "installed_version": "2.4.1",
            "ocsid": "166051",
            "types": ""
        },
		"files_trashbin": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "filesystem"
        },
        "files_versions": {
            "enabled": "yes",
            "installed_version": "1.5.0",
            "types": "filesystem"
        },
        "files_videoplayer": {
            "enabled": "yes",
            "installed_version": "1.1.0",
            "types": ""
        },
        "firstrunwizard": {
            "enabled": "yes",
            "installed_version": "2.1",
            "types": "logging"
        },
        "gallery": {
            "enabled": "yes",
            "installed_version": "17.0.0",
            "types": ""
        },
        "groupfolders": {
            "enabled": "yes",
            "installed_version": "1.1.0",
            "types": "filesystem"
        },
        "keeweb": {
            "enabled": "yes",
            "installed_version": "0.4.0",
            "types": ""
        },
        "logreader": {
            "enabled": "yes",
            "installed_version": "2.0.0",
            "ocsid": "170871",
            "relativedates": "1",
            "types": ""
        },
        "lookup_server_connector": {
            "enabled": "yes",
            "installed_version": "1.0.0",
            "types": "authentication"
        },
        "music": {
            "enabled": "no",
            "installed_version": "0.5.5",
            "types": "filesystem"
        },
        "nextcloud_announcements": {
            "enabled": "yes",
            "installed_version": "1.1",
            "pub_date": "Sat, 10 Dec 2016 00:00:00 +0100",
            "types": "logging"
        },
        "notifications": {
            "enabled": "yes",
            "installed_version": "2.0.0",
            "types": "logging"
		},
        "oauth2": {
            "enabled": "yes",
            "installed_version": "1.0.5",
            "types": "authentication"
        },
        "ocdownloader": {
            "enabled": "no",
            "installed_version": "1.5.5",
            "types": ""
        },
        "ownbackup": {
            "enabled": "no",
            "installed_version": "17.5.0",
            "types": ""
        },
        "password_policy": {
            "enabled": "yes",
            "installed_version": "1.2.2",
            "types": ""
        },
        "provisioning_api": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "prevent_group_restriction"
        },
        "quota_warning": {
            "enabled": "yes",
            "installed_version": "1.1.1",
            "types": "filesystem"
        },
        "ransomware_protection": {
            "enabled": "yes",
            "installed_version": "1.0.5",
            "types": "logging"
        },
        "serverinfo": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": ""
        },
        "sharebymail": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "filesystem"
        },
        "survey_client": {
            "enabled": "yes",
            "installed_version": "1.0.0",
            "types": ""
        },
        "systemtags": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "logging"
        },
        "tasks": {
            "enabled": "yes",
            "installed_version": "0.9.5",
            "types": ""
        },
        "theming": {
            "cachebuster": "2",
            "enabled": "yes",
            "installed_version": "1.3.0",
            "name": "DutchCloud",
            "types": "logging",
            "url": "https:\/\/dutches.ddns.net"
        },
        "twofactor_backupcodes": {
            "enabled": "yes",
            "installed_version": "1.1.1",
            "types": ""
        },
        "updatenotification": {
            "audioplayer": "2.2.3",
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "",
            "update_check_errors": "0"
        },
        "workflowengine": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "filesystem"
        }
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
No
Are you using encryption: yes/no
No
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

Client configuration

Browser:
Using WebDav.

Operating system:
Windows 10

Logs

Web server error log

Web server error log 
Error log is empty.

Nextcloud log (data/nextcloud.log)

Nextcloud log 
No errors.
@Flying--Dutchman
Copy link
Author

Flying--Dutchman commented Jan 19, 2018 •
edited
Loading

It looks like nextcloud does alot of stuff in the background, when the actually 'upload' is done.
There is a pause after each file, which increases depending on the size of the uploaded file.

Why does the action after uploading a ~500mb file take multiple minutes?
Neither CPU nor RAM is at full capacity.

Copy process:
image

CPU and RAM usage of the server:
image

@nickvergessen
Copy link
Member

Well check the Disk IO, after the upload is "finished" the uploaded chunks need to be assembled to the complete file again.

@Flying--Dutchman
Copy link
Author

Correct, the nginx worker writes with 1.500kb to the disk after the upload. Does Nextcloud have to wait for this operation? Can't it be done in the background, asynchronously?

@MorrisJobke
Copy link
Member

Correct, the nginx worker writes with 1.500kb to the disk after the upload. Does Nextcloud have to wait for this operation? Can't it be done in the background, asynchronously?

Sadly not, otherwise the request can't return with a success code (Something like "Created"), but this is required by webdav. We would need to say: "yes we received it, but it is in an intermediate state", which is not available in webdav :/ So this needs to be done synchronously. Sorry for the bad news here.

@nextcloud-bot nextcloud-bot mentioned this issue Jul 14, 2018
Closed
@nextcloud-bot nextcloud-bot mentioned this issue Nov 6, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nickvergessen @MorrisJobke @Flying--Dutchman