Dynamic chunk sizing for faster upload of big files

[svenlange2]

Expected behaviour

Big files should be easy to upload and should not stress server nor client as there should be moderate SQL activity (in contrast to notorious small file upload slowdown problems)

Actual behaviour

In Android sync client big files (ten or hundreds of megabytes) are generally slow to upload. For example a 200 MB video starts off nicely at somewhat below connection limited speed but drops in about 1 minutes gradually to ~30% of the speed and further down. My server is powerful (4 cores, 4GB RAM etc.) and connection to the server is over wifi (real TCP throughput ~40 Mbit). Example: upload starts at ~25 Mbit/s. In 60 sec its down to 10 Mbit and in 120s its ~5-6 mbit (given that the file is big enough).

Steps to reproduce

1. Upload at least 100 MB file over a connection where it takes > 1 min to upload.
2. Observe speed at network interface (Android or server side)

Can this problem be reproduced with the official owncloud server?
(url: https://demo.owncloud.org, user: test, password: test)
Cannot test - my outside uplink is too slow. I am using server ownCloud 8.2.3 (beta) myself currently. Similar problems persisted with 8.1.

SOLUTION

I recompiled the 1.9.1 source with one modification in file /owncloud-android-library/src/com/owncloud/android/lib/resources/files/ChunkedUploadRemoteFileOperation.java

public static final long CHUNK_SIZE = 1024000;

public static final long CHUNK_SIZE = 10240000; // added one "0", ~10M

RESULT

Enormous speedup. My uploads go at ~40 Mbit/s and speed is not dropping !
So the problem is the CHUNK size that fragments the upload, and creates obviously comparably more SQL activity at server.

If you are interested in recompiled apk, ask me: sven(at)omapilv.ee

RECOMMENDATION for FIX

Add possibility to change chunk size to 10M in settings. Choise is needed as some people may find it convenient to use 1M chunk for unstable connections (discussed in some threads). In wifi its usually not needed.

Environment data

Android version:
5.1
Device model:
One Plus X
Stock or customized system:
Stock
ownCloud app version:
1.9.1
ownCloud server version:
8.2.3

[tobiasKaminsky]

I am not that into /core but why should there such a high load of SQL when using smaller chunks?
200Mb might be 200 chunks, but I doubt that the chunk is stored in SQL, rather a checksum and that should not produce such a load...
Can you see where the high load is?

[svenlange2]

Hi!

I checked the load. With 1M chunks the server (4GB RAM, 4 core Celeron)
has ~35% load. Main load was generated by php-fpm processes and httpd.
The server runs Synology DSM5.2 (linux). I tried various MarianDB
optimizations from forums before to reduce SQL load by increasing
various buffers, but it did not change the behavior. I dont know the
relationship between SQL and php-fpm exactly, but my guess is that there
is some fixed amount of operations that the server must do in order to
receive every chunk and they sort of built up if reception is speedy
enough. At 10Mbit/s it would translate to 1 chunk per second x3 parallel
connections(server default). After I increased chunk size to 10M the
load dropped to 15% and speed as mentioned does not drop anymore and is
at TCP link maximum. It feels like the reception process is able to
handle all tasks before the next chunk is received (at 40 Mbit/s I get a
chunk in 2-3 sec).

I also think that the php+SQL tasks performed with every chunk are
concentrated at right after the end of a file chunk reception and in the
beginning (multiplied by 3 parallel reception connections (default)).
Now if those take time and the reception does not start async to receive
another file chunk then the logic actually delays reception. I dont know
the logic of the server thou so, dont take that seriously - just a guess.

The bottom line is that more such delays in a given timeframe the more
delayed is the reception at high speed links. But for videos (I shoot
4K) anything below 4g or wifi would be useless anyway.

I hope the info is helpful in some way.

Sven

On 04/21/2016 08:09 PM, Tobias Kaminsky wrote:

I am not that into /core but why should there such a high load of SQL
when using smaller chunks?
200Mb might be 200 chunks, but I doubt that the chunk is stored in
SQL, rather a checksum and that should not produce such a load...
Can you see where the high load is?

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#1620 (comment)

[svenlange2]

another observation: In Linux desktop client the same speed dropping is evident at gigabit link. When I add a big file (500MB) to my sync folder and the client starts to upload it to server the upload starts lets say at 40 MB/s and drops to ~15 MB/s at the end of upload. As the speed is still high enough I dont bother but the same logic seems to affect it somehow. Chunk size default 1M I guess as I have stock client.

Edit: Server load topped ~60% in this upload. Maybe it is server limited this time as changing the chunk size to 10M or even 100M with "export OWNCLOUD_CHUNK_SIZE" + client restart did not fix the slowdown entirely - got 20MB/s in the end of upload.

Edit2: Problem source is probably client software speed (and platform its running on). Namely over wifi on desktop I get constant 15 MB/s upload (300 Mbit/s link). So the client is fast enough for that TCP link and server is able to handle. Over gigabit the slowdown is probably caused by server as on client side no significant load is observable.

Android phone is not fast enough to upload 1M chunks over 40Mbit/s TCP link and crawls. With 10M chunks speed is ok for 40 Mbit/s - no degradation.

[LukeOwlclaw]

Sounds a lot like: owncloud/client#4354
They are using dynamic chunk size now.

[xeros]

It seems like it's not fixed yet (still getting it on 2.1.0, but I can't use 2.1.1/2.1.2/2.1.3 due to "App not responding to touch events #1799" bug in android client versions past 2.1.0).
The problem is that I'm usually uploading recorded videos from phone (300MB-2GB), which get limited in upload to around 550kB/s droping to 0 for 1-2 seconds on each chunk (on 150/300MBit 802.11n WIFI network, even when I put the phone 10cm to router (ownCloud server on Banana Pro) antenna).
Both MySQL database & ownCloud temp & upload space are on fast SATA disk and CPU is not busy by storage I/O, but php5-fpm uses 50%-90% of single CPU core and mysqld 2%-8% (it's dual core system with still 110-130% CPU core resources still available when ownCloud Android client is uploadin files).
By using other upload services I get more than 30x times better speed than ownCloud when uploading my recorded videos from phone to this Banana Pro server.
I'm pretty sure that I could get better results when I could increase chunk size (I see that it's currently 1.024.000bytes (a bit less than 1MB)), also looks like PHP code may not be optimal.

Edit:
Phone is also fast (8-core @ 1.6GHz MT6592, with fast internal flash (getting more than 100MB/s in read and write) and microSDXC card (64GB Kingston U3 R:90/W:85).

[davivel]

Let me update the title for better visibility in list of issues. Thanks.

Related: owncloud/client#4875

[michaelstingl]

Will be solved with with support for TUS.io upload protocol (#2978)