[ocis][clients] Large File Uploads

[micbar]

Future

• Use TUS io protocol www.tus.io

Use Cases

• Large File Uploads
• Instable connections
• middlewares are interfering

Nasty middleware example:
PUT > 100MB will fail:
https://demo.shniq.cloud/
Cloudflare limits upload size (HTTP POST request size) per plan type:
100MB Free and Pro
(Source: https://support.cloudflare.com/hc/en-us/articles/200172516#h_51422705-42d0-450d-8eb1-5321dcadb5bc)

Cern

• Needs to be decided it CERN and Clients will stick to Chunking V1 and go directly to Tus (V3)

oCIS

• OC-DAV handler will forward tus requests
• expose capabilities: Expose TUS capabilities through capabilities API ocis-reva#177

Reva changes

• [WIP] Use tus for uploads (and support range requests for downloads?) cs3org/reva#286
• COPY regression with TUS: Webdav COPY must use TUS for its internal uploads ocis-reva#173
• preserve mtime: Preserve mtime after TUS upload ocis-reva#174 (to discuss)
• issue with creation-with-upload: TUS creation-with-upload returns 400 using OC storage cs3org/reva#710

Phoenix

• basic TUS impl
• creation-with-upload support: Add TUS creation-with-upload support web#3436

Start next sprint with TUS client https://github.com/owncloud/enterprise/issues/3901

• Server Side Data Service
• Oc-DAV Service
• JS Client added to phoenix/sdk: File uploads with chunking and resumable web#67
• Implement the js client

Effort: M
Complexity: S/M

Desktop

• Start with TUS for 2.8

iOS

• TUS implementaition will start as soon as Phoenix POC is tested

Android...

[micbar]

@michaelstingl @butonic @felixboehm

[PVince81]

We might want to clarify that the scope of this only applies to authenticated uploads, not public link nor file drop. In OC 10 we decided against chunking for anonymous access due to the possibility of spamming the upload cache with random chunks by unauthenticated users.

[PVince81]

Some tech info about the curl requests for a simple "two-request" based upload which can be useful for clients:

1. Curl for initiating an upload:

curl -D - 'https://localhost:9200/remote.php/dav/files/einstein/' \
  -X 'POST' \
  -u einstein:relativity \
  -H 'Tus-Resumable: 1.0.0' \
  -H 'Upload-Length: 10000' \
  -H 'Upload-Metadata: filename bG1tcy0xLjIuMC1yYzgtbGludXgteDg2XzY0LkFwcEltYWdl,filetype YXBwbGljYXRpb24vdm5kLmFwcGltYWdl,size OTgxNjg1MjA=,mtime MTU1MjIzOTI1MC4wNDg=' \
  --insecure
HTTP/1.1 201 Created
Access-Control-Allow-Headers: Tus-Resumable, Upload-Length, Upload-Metadata, If-Match
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Tus-Resumable, Location
Content-Length: 0
Date: Mon, 27 Apr 2020 13:52:17 GMT
Location: http://localhost:9164/data/3b0ba66e-4745-437e-9410-6d9a7ee06eee
Tus-Resumable: 1.0.0
Vary: Origin
X-Access-Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNTg3OTk5MTM3LCJpYXQiOjE1ODc5OTU1MzcsImlzcyI6Imh0dHBzOi8vbG9jYWxob3N0OjkyMDAiLCJ1c2VyIjp7ImlkIjp7ImlkcCI6Imh0dHBzOi8vbG9jYWxob3N0OjkyMDAiLCJvcGFxdWVfaWQiOiJlaW5zdGVpbiJ9LCJ1c2VybmFtZSI6ImVpbnN0ZWluIiwibWFpbCI6ImVpbnN0ZWluQGV4YW1wbGUub3JnIiwiZGlzcGxheV9uYW1lIjoiRWluc3RlaW4ifX0.etJPHNQHXOJnZY88b3ZL7HZ55_PhXvKBA6Lm2tjS-LY

=> you can replace username/password with an authorization + bearer token instead

2. Then send the data there to the location got in the "Location" header:

curl -D - -k 'http://localhost:9164/data/3b0ba66e-4745-437e-9410-6d9a7ee06eee' \
  -X 'PATCH' \
  -u einstein:relativity \
  -H 'Connection: keep-alive' \
  -H 'Tus-Resumable: 1.0.0' \
  -H 'Upload-Offset: 0' \
  -H 'Content-Type: application/offset+octet-stream' \
  -d "abcdefghijklmno"  
HTTP/1.1 204 No Content
Tus-Resumable: 1.0.0
Upload-Offset: 15
X-Access-Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNTg3OTk5NjIxLCJpYXQiOjE1ODc5OTYwMjEsImlzcyI6Imh0dHBzOi8vbG9jYWxob3N0OjkyMDAiLCJ1c2VyIjp7ImlkIjp7ImlkcCI6Imh0dHBzOi8vbG9jYWxob3N0OjkyMDAiLCJvcGFxdWVfaWQiOiJlaW5zdGVpbiJ9LCJ1c2VybmFtZSI6ImVpbnN0ZWluIiwibWFpbCI6ImVpbnN0ZWluQGV4YW1wbGUub3JnIiwiZGlzcGxheV9uYW1lIjoiRWluc3RlaW4ifX0.ES-3q4Cf1tEbF_umOIzzXbLcLujeAHHo4bkUtI6PwAg
X-Content-Type-Options: nosniff
Date: Mon, 27 Apr 2020 14:00:21 GMT

Since we only uploaded a few bytes, the offset moved to 15

Replace "abcdefghijklmno" with the file data. (I didn't manage to and -F uses multipart which mangles the content-type)

[PVince81]

I've updated the checkboxes and added a few more as I continue finding small issues

[PVince81]

Matching Jira ticket https://jira.owncloud.com/browse/OCIS-9 was closed.