Improve caching & preloading for better performance [$25]

[jancborchardt]

Often when I use the search and go through mails (mostly deleting them), it takes a lot of time to load the next mail after I delete one. It would be cool if when you open a view, or make a search, the content of the batch of mails is actually preloaded.

---

There is a $25 open bounty on this issue. Add to the bounty at Bountysource.

[ChristophWurst]

Agreed. Still, we have to work on the server-side implementation too, I have the feeling we could do a lot better there in terms of performance.

[ChristophWurst]

To potential bounty hunters:

1. Let us know when you plan working on this
2. Let's discuss the implementation together
3. …
4. Profit

[RNCTX]

107.202.13.161 - - [02/Apr/2020:12:47:14 -0500] "GET /apps/mail/ HTTP/2.0" 200 8587 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"
127.0.0.1 - - [02/Apr/2020:12:47:14 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
107.202.13.161 - - [02/Apr/2020:12:47:14 -0500] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/2.0" 200 74 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"
127.0.0.1 - - [02/Apr/2020:12:47:15 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:16 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:17 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:18 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:19 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:20 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:21 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:22 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:23 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:24 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:25 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
107.202.13.161 - - [02/Apr/2020:12:47:26 -0500] "GET /apps/mail/api/accounts/1/folders/SU5CT1g%3D/messages HTTP/2.0" 200 2164 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"
127.0.0.1 - - [02/Apr/2020:12:47:26 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:27 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
107.202.13.161 - - [02/Apr/2020:12:47:28 -0500] "GET /apps/mail/api/accounts/1/folders/SU5CT1g%3D/messages/35530 HTTP/2.0" 200 368 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"
127.0.0.1 - - [02/Apr/2020:12:47:28 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:29 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:30 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"
127.0.0.1 - - [02/Apr/2020:12:47:31 -0500] "GET /stub_status HTTP/1.1" 200 104 "-" "-"

This is on a gmail account with ~9000 messages. The above netdata log is immediately after clicking the mail tab in the header, and waiting until the load finishes. Current nextcloud (18.0.3), without any other users on the system, just the admin logged in on a fresh install, on a 2 CPU / 4GB RAM linode VPS. OnlyOffice is installed and Postgres is tuned for 2gb of available memory. Nginx for the webserver, on which nextcloud is installed directly, no reverse proxy.

[RNCTX]

Just installed the beta linked in the help forums, after initial caching the performance is GREATLY improved.

[ChristophWurst]

Just installed the beta linked in the help forums, after initial caching the performance is GREATLY improved.

Thanks so much. I really appreciate the feedback!

[ChristophWurst]

I guess this feature is mostly done via #1956 and #2064

[ChristophWurst]

This is on a gmail account with ~9000 messages.

Was that with 1.1.4 or the beta?

[RNCTX]

That was on 1.1.4.

Same test on the beta here, on the same Nextcloud server and the same GMail account with ~9000 emails in it:

And the netdata log from that time frame:

[ChristophWurst]

So if I interpret these measurements correctly, the load was decreased significantly 🎆

[RNCTX]

Yeah I know they're hard to see scaled down but the moving meters are, left to right... disk I/O, CPU, and network traffic.

After the initial cache of existing mail, what was 50-55% CPU on initial page load went down to 4%, and peak bandwidth usage during the cache build was comfortably handled in 2-3Mbps chunks for a few seconds each, it seemed.

It's not quite as snappy as the Gmail web interface but close ;).

[ChristophWurst]

It's not quite as snappy as the Gmail web interface but close ;).

haha guess they have like 50x as many people working on their app but fair enough 😁

[RNCTX]

Yeah, I meant that as a compliment. I'm sure if you could cache every single image on the internet like Google can, you could probably find 250ms here and there too!

[ChristophWurst]

Btw you're famous now: https://nextcloud.com/blog/mail-1-3-introduces-mail-cache-improved-signatures-link-shares-mark-all-read-and-more/ :)

[ChristophWurst]

Done via #1956 and #2064

[ChristophWurst]

hey @johanlives the ticket at https://www.bountysource.com/issues/42324763-improve-caching-preloading-for-better-performance does not close. can you help?