cannot upload large file to ocis server #267

jnweiger · 2020-05-12T01:00:09Z

Test setup via https://gitea.owncloud.services/jw/hetzner/src/branch/master/make_ocis_test.sh

cd ocis
make generate build
./bin/ocis server

[EDIT: this can be reproduced with Phoenix. See further below
The client scenario may have been spoiled by e.g. https://github.com/owncloud/client/issues/7863]

client connected via https://localhost:9200

When creating a folder Music on the local filesystem, the folder also gets created on the server. OK.
When copying a MP3 file of ca 10MB into the folder Music, the sync to the server fails:

when uploading the same MP3 file of ca 10 MB via the Web UI, the upload fails:

A small 40k bytes mp3 file can be uploaded without problem. both via Web ui and desktop client.

The text was updated successfully, but these errors were encountered:

jnweiger · 2020-05-12T01:00:49Z

Server console excerpt, when trying with the web interface:

2020-05-12T02:58:04+02:00 INF access token is already provided pkg=rhttp service=reva traceid=6e3cb1bc7272cb5c8b09b02f4535e028
2020-05-12T02:58:14+02:00 ERR error doing http request error="Put \"http://localhost:9156/data/Music/Polymorphia_for_48_Penderezki.mp3\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" pkg=rhttp service=reva traceid=6e3cb1bc7272cb5c8b09b02f4535e028
2020-05-12T02:58:14+02:00 ERR http end="12/May/2020:02:58:14 +0200" host=127.0.0.1 method=PUT pkg=rhttp proto=HTTP/1.1 service=reva size=0 start="12/May/2020:02:58:04 +0200" status=500 time_ns=10020847183 traceid=6e3cb1bc7272cb5c8b09b02f4535e028 uri=/remote.php/webdav/Music/Polymorphia_for_48_Penderezki.mp3 url=/remote.php/webdav/Music/Polymorphia_for_48_Penderezki.mp3
2020-05-12T02:58:14+02:00 ERR error uploading file error="ocfs: error writing to tmp file /var/tmp/reva/data/marie/files/Music/._reva_atomic_upload721223429: unexpected EOF" pkg=rhttp service=reva traceid=6e3cb1bc7272cb5c8b09b02f4535e028
2020-05-12T02:58:14+02:00 ERR http end="12/May/2020:02:58:14 +0200" host=127.0.0.1 method=PUT pkg=rhttp proto=HTTP/1.1 service=reva size=0 start="12/May/2020:02:58:04 +0200" status=500 time_ns=10002981106 traceid=6e3cb1bc7272cb5c8b09b02f4535e028 uri=/data/Music/Polymorphia_for_48_Penderezki.mp3 url=/data/Music/Polymorphia_for_48_Penderezki.mp3
2020-05-12T02:58:17+02:00 WRN core access token not set pkg=rhttp service=reva traceid=92855e85e51da84001fcc2a59e593103
2020-05-12T02:58:17+02:00 INF unary code=OK end="12/May/2020:02:58:17 +0200" from=tcp://127.0.0.1:50554 pkg=rgrpc service=reva start="12/May/2020:02:58:17 +0200" time_ns=86023 traceid=92855e85e51da84001fcc2a59e593103 uri=/cs3.auth.registry.v1beta1.RegistryAPI/GetAuthProvider user-agent=grpc-go/1.26.0
2020-05-12T02:58:17+02:00 WRN root/go/pkg/mod/github.com/cs3org/reva@v0.1.1-0.20200427161359-c1549a8110eb/internal/grpc/services/authprovider/authprovider.go:107 > error authenticating user error="authsvc: error in Authenticate: oidc: error getting userinfo: +401 Unauthorized: " pkg=rgrpc service=reva traceid=92855e85e51da84001fcc2a59e593103
2020-05-12T02:58:17+02:00 INF unary code=OK end="12/May/2020:02:58:17 +0200" from=tcp://127.0.0.1:57230 pkg=rgrpc service=reva start="12/May/2020:02:58:17 +0200" time_ns=10688942 traceid=92855e85e51da84001fcc2a59e593103 uri=/cs3.auth.provider.v1beta1.ProviderAPI/Authenticate user-agent=grpc-go/1.26.0
2020-05-12T02:58:17+02:00 ERR error authenticating credentials to auth provider for type: bearer error="gateway: grpc failed with code CODE_UNAUTHENTICATED" pkg=rgrpc service=reva traceid=92855e85e51da84001fcc2a59e593103
2020-05-12T02:58:17+02:00 WRN root/go/pkg/mod/github.com/cs3org/reva@v0.1.1-0.20200427161359-c1549a8110eb/internal/grpc/services/gateway/authprovider.go:66 >  error="gateway: grpc failed with code CODE_UNAUTHENTICATED" pkg=rgrpc service=reva traceid=92855e85e51da84001fcc2a59e593103
2020-05-12T02:58:17+02:00 INF unary code=OK end="12/May/2020:02:58:17 +0200" from=tcp://127.0.0.1:50552 pkg=rgrpc service=reva start="12/May/2020:02:58:17 +0200" time_ns=13814082 traceid=92855e85e51da84001fcc2a59e593103 uri=/cs3.gateway.v1beta1.GatewayAPI/Authenticate user-agent=grpc-go/1.26.0
2020-05-12T02:58:17+02:00 ERR error generating access token from credentials error="auth: grpc failed with code CODE_UNAUTHENTICATED" pkg=rhttp service=reva traceid=92855e85e51da84001fcc2a59e593103
2020-05-12T02:58:17+02:00 WRN http end="12/May/2020:02:58:17 +0200" host=127.0.0.1 method=PROPFIND pkg=rhttp proto=HTTP/1.1 service=reva size=0 start="12/May/2020:02:58:17 +0200" status=401 time_ns=15948035 traceid=92855e85e51da84001fcc2a59e593103 uri=/remote.php/dav/files/einstein/ url=/remote.php/dav/files/einstein/

jnweiger · 2020-05-12T10:15:47Z

Also seen with the copose setup

    git clone https://github.com/owncloud-docker/compose-playground.git
    cd compose-playground/compose/ocis
    sed -i -e 's/your-url/$ipaddr/g' config/identifier-registration.yml
    echo >> .env OCIS_BASE_URL=$ipaddr
    echo >> .env OCIS_HTTP_PORT=9200
    echo >> .env OCIS_DOCKER_TAG=1.0.0-beta4
    docker-compose -f ocis.yml -f ../cache/redis-ocis.yml up

Phoenix web gui shows a nice progress bar for the 10MB file upload, that progrresses up to 27 percent, then resets, and repeats for several attempts.

individual-it · 2020-05-12T10:21:05Z

I'm surprised that it does not work, because there is an API test that uploads and downloads a big file
https://github.com/owncloud/core/blob/fc0c5ba5716fad56b43297a711b1ad909d506660/tests/acceptance/features/apiWebdavOperations/downloadFile.feature#L32
Does the client try to do chunking?

individual-it · 2020-05-12T10:46:44Z

The issue seems to be triggered if the file cannot be uploaded within a time-period. To force it throttle the connection
https://jankaritech.ocloud.de/index.php/s/Lke4pazrOEIPabE

michaelstingl · 2020-05-12T13:56:58Z

Multi-dimensional problem:

oCIS announces chunking

% curl -s -k -u einstein:relativity 'https://ocis-latest.owncloud.com/ocs/v1.php/cloud/capabilities?format=json' | jq . | grep -i -e  chunk -e tus | grep -i -v status

          "bigfilechunking": false,
          "chunking": "1.0",
          "chunkingParallelUploadDisabled": false

Also see: #203

Chunking V1 and Chunking NG aren't working in oCIS. (Chunking V1 broken #195 and oCIS triggers chunking NG in desktop client #194)
Desktop client partially ignores capabilities and performs Chunking V1 in case of absence

Desktop issue for 2.7: owncloud/client#7862

(Similar for Android, fix planned for 2.15)

jnweiger · 2020-05-12T14:05:36Z

More dimensions to add to that multidimensionality:

web UI is also affected: Is Phoenix also doing chunking?

jnweiger · 2020-05-12T14:14:04Z

Cannot test this with ocis on eos, due to #265

michaelstingl · 2020-05-13T08:15:34Z

web UI is also affected: Is Phoenix also doing chunking?

No, I don't hink so. Better handled in new issue I'd say…

PVince81 · 2020-05-13T08:17:22Z

there is likely a timeout issue in OCIS.
once when testing we noticed that downloaded files got truncated after 10 seconds or so, might also affect uploads

@butonic is this a deployment thing or something we need to change in our default configs ?

jnweiger · 2020-05-13T08:28:42Z

web UI is also affected: Is Phoenix also doing chunking?

No, I don't hink so. Better handled in new issue I'd say…

Sorry for this messy issue. When I started, I thought small files had uploaded successfully with the client. So I concluded that it is related to the size. Currently I cannot upload any files, small or large with a desktop client. So forget my ramblings about client chunking and ignore the misleading client screenshots above. The client issue to follow up is #265

This messy issue here should focus on large files and timeouts. That is already reproducable with phoenix. No desktop client involved.

butonic · 2020-06-02T14:49:28Z

the timeout part of the problem is tracked in #120

jnweiger · 2020-07-15T15:00:07Z

Re-tested with client-2.7.0daily20200708 vs docker image owncloud/ocis:1.0.0-beta8 (plain owncloud storage)

no problems with small files
transient issues with many small files (see also File upload via desktop client needs retries #309)
bad behviour with large files (see https://github.com/owncloud/ocis/issues/385)

As far as I can see the basic issue is fixed with beta8, more detailled issues are tracked elsewhere.
I am fine with closing here, if chunking works. Not tested yet.

jnweiger · 2020-09-14T23:10:32Z

retested with

ocis-1.0.0-rc1 via https://github.com/owncloud-docker/compose-playground/blob/master/examples/hetzner-deploy/make_ocis_eos_compose_test.sh
and desktop daily (20200915) client ownCloud version 2.7.0 (build 2106) on Ubuntu 20.04
and firefox on an ubuntu desktop

Version details ocis-1.0.0-rc1

OCIS_VERSION:         v1.0.0-rc1
ocis --version:       ocis version 199267c^M
git log:              commit 199267c13e916aa8de027a81dbf6733af820369f (HEAD, tag: refs/tags/v1.0.0-rc1)
eos --version:        EOS 4.6.5 (2019)^M
xrootd -v:            v4.11.0^M
bin/ocis contains:
        owncloud/ocis-accounts@v0.4.2-0.20200901074457-6a27781a2741
        owncloud/ocis-glauth@v0.5.1-0.20200731165959-1081de7c60f1
        owncloud/ocis-graph-explorer@v0.0.0-20200210111049-017eeb40dc0c
        owncloud/ocis-graph@v0.0.0-20200318175820-9a5a6e029db7
        owncloud/ocis-hello@v0.1.0-alpha1.0.20200828085053-37fcf3c8f853
        owncloud/ocis-konnectd@v0.3.2
        owncloud/ocis-migration@v0.2.0
        owncloud/ocis-ocs@v0.3.1
        owncloud/ocis-phoenix@v0.13.0
        owncloud/ocis-pkg/v2@v2.4.1-0.20200828095914-d3b859484b2b
        owncloud/ocis-proxy@v0.7.1-0.20200904132806-fcceec602fcb
        owncloud/ocis-reva@v0.13.0
        owncloud/ocis-settings@v0.3.2-0.20200902094647-35dc3aeaba78
        owncloud/ocis-store@v0.1.1
        owncloud/ocis-thumbnails@v0.3.0
        owncloud/ocis-webdav@v0.1.1

Upload of an 5.5 GB file works fine on RC1. Both, Phoenix and desktop.

[Tests-Only] Fetch API testrunner from specific branch and commit

jnweiger changed the title ~~cannot upload large mp3 file to ocis server~~ cannot upload large file to ocis server May 12, 2020

butonic mentioned this issue Jul 1, 2020

add new config options for the http client owncloud/ocis-reva#330

Merged

3 tasks

jnweiger closed this as completed Sep 16, 2020

refs pushed a commit that referenced this issue Sep 18, 2020

Merge pull request #267 from owncloud/apiTests-from-branch-commit

b443db2

[Tests-Only] Fetch API testrunner from specific branch and commit

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cannot upload large file to ocis server #267

cannot upload large file to ocis server #267

jnweiger commented May 12, 2020 •

edited

Loading

jnweiger commented May 12, 2020

jnweiger commented May 12, 2020

individual-it commented May 12, 2020

individual-it commented May 12, 2020

michaelstingl commented May 12, 2020 •

edited by jnweiger

Loading

jnweiger commented May 12, 2020

jnweiger commented May 12, 2020

michaelstingl commented May 13, 2020

PVince81 commented May 13, 2020

jnweiger commented May 13, 2020 •

edited

Loading

butonic commented Jun 2, 2020 •

edited

Loading

jnweiger commented Jul 15, 2020

jnweiger commented Sep 14, 2020 •

edited

Loading

cannot upload large file to ocis server #267

cannot upload large file to ocis server #267

Comments

jnweiger commented May 12, 2020 • edited Loading

jnweiger commented May 12, 2020

jnweiger commented May 12, 2020

individual-it commented May 12, 2020

individual-it commented May 12, 2020

michaelstingl commented May 12, 2020 • edited by jnweiger Loading

jnweiger commented May 12, 2020

jnweiger commented May 12, 2020

michaelstingl commented May 13, 2020

PVince81 commented May 13, 2020

jnweiger commented May 13, 2020 • edited Loading

butonic commented Jun 2, 2020 • edited Loading

jnweiger commented Jul 15, 2020

Re-tested with client-2.7.0daily20200708 vs docker image owncloud/ocis:1.0.0-beta8 (plain owncloud storage)

jnweiger commented Sep 14, 2020 • edited Loading

retested with

jnweiger commented May 12, 2020 •

edited

Loading

michaelstingl commented May 12, 2020 •

edited by jnweiger

Loading

jnweiger commented May 13, 2020 •

edited

Loading

butonic commented Jun 2, 2020 •

edited

Loading

jnweiger commented Sep 14, 2020 •

edited

Loading