Drag & Drop large file not copied when the file exceed 60M

[lmcbout]

Drag & Drop large file. Follow up on issue #868
Tried on Chrome and Firefox: was able to transfer file 60M and the md5sum is ok, but when I try to drag a file of 80M, the browser stops with the following error:
main.js:816 Uncaught (in promise) Error: Connection got disposed.
at Object.dispose (main.js:816)
at connection.ts:14
at CallbackList.invoke (events.js:71)
at Emitter.fire (events.js:135)
at closeHandler (main.js:212)
at CallbackList.invoke (events.js:71)
at Emitter.fire (events.js:135)
at WebSocketMessageReader.AbstractMessageReader.fireClose (messageReader.js:126)
at WebSocketMessageReader.fireClose (reader.ts:72)
at reader.ts:31

[lmcbout]

Some discussion took place with a similar problem:
https://stackoverflow.com/questions/6135427/increasing-the-maximum-post-size

[akosyakov]

Is it the web socket limitation?

[lmcbout]

From the link above, it seems to be a limitation on the server side. With Apache, we can set the size either in .htaccess like ; Maximum allowed size for uploaded files.
upload_max_filesize = 40M
Right now, I haven't found were in Theia we have a limitation. I tried with a file of 60M and it transferred correctly, but failed with a bigger file

[hexa00]

@lmcbout but this is using websocket no ? it's not an http post.

So these limitations should not apply

[lmcbout]

@hexa00 We have a limitation on something that prevent large file to be uploaded, As I said earlier, I don't know where, but the server does not allow us in Theia to have file larger than ~60M.
On the link above, they had a limitation even smaller than the one in Theia.

[lmcbout]

It is a limitation outside of the Drag & Drop code otherwise, no files/folders would be transferred

[lmcbout]

While debugging, I noticed that the large files get split into smaller packets ~ 10M and is sent from the client to the server

[hexa00]

@lmcbout I just mean that upload_max_filesize , applies to http post and not websockets.

When you're sending something though websocket the server has no idea if it's a file upload.. it's just data.

[lmcbout]

@hexa00 This is my understanding as well :)

[lmcbout]

More investigation about the large file size in Theia:
Using Chrome/Firefox : same result
Able to load many large files (78.625M or less ) up to 750M in Theia ===> no problem
When a single file file reach 78.750M, then we get an error. If you try to load another large file which is 78.625M or less, it loads ok even after the error for the larger file.
Error reported with large file (78.750M or higher):
main.js:816 Uncaught (in promise) Error: Connection got disposed.
at Object.dispose (main.js:816)
at connection.ts:14
at CallbackList.invoke (events.js:71)
at Emitter.fire (events.js:135)
at closeHandler (main.js:212)
at CallbackList.invoke (events.js:71)
at Emitter.fire (events.js:135)
at WebSocketMessageReader.AbstractMessageReader.fireClose (messageReader.js:126)
at WebSocketMessageReader.fireClose (reader.ts:72)
at reader.ts:31

[hexa00]

This is a problem in the vscode's json-rpc library because send closes the connection if you try to send with a full socket buffer see: https://developer.mozilla.org/en-US/docs/Web/API/WebSocket

send()
Enqueues the specified data to be transmitted to the server over the WebSocket connection, increasing the value of bufferedAmount by the number of bytes needed to contain the data. If the data can't be sent (for example, because it needs to be buffered but the buffer is full), the socket is closed automatically.

We need to throttle send so that the connection isn't lost.

@svenefftinge could typefox work on this ?