Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Performance] Reuse preview directory listing #35129

Merged
merged 8 commits into from
Apr 17, 2023

Conversation

Glandos
Copy link
Contributor

@Glandos Glandos commented Nov 13, 2022

Superseed #34910

Use directory listing in both functions to gain 25% speed on run where every preview already exist.

I went from "./occ preview:generate-all" from 159 seconds down to 17 seconds on a directory full of images with already generated previews.

Copy link
Contributor

@artonge artonge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I right that this PR "only" moves the getDirectoryListing call from getSmallImagePreview to generatePreviews?
All other code changes are to:

  • pass previewFiles to all the methods
  • reuse some logic to make getSmallImagePreview smaller with getCachedPreview and generateProviderPreview

@Glandos
Copy link
Contributor Author

Glandos commented Dec 20, 2022

Am I right that this PR "only" moves the getDirectoryListing call from getSmallImagePreview to generatePreviews? All other code changes are to:

* pass `previewFiles` to all the methods

* reuse some logic to make `getSmallImagePreview` smaller with `getCachedPreview` and `generateProviderPreview`

Yes, the rationale is that calling getDirectoryListing multiple times is slow and useless, since the directory won't change between two invocation, given the proximity of the code. So instead of passing previewDirectory, the list, output from getDirectoryListing($previewDirectory) is passed along.

I've noticed that getSmallImagePreview was added with Imaginary support, but the code was duplicated. Since my modifications modify the code, I gathered them in one function.

@PVince81 PVince81 requested a review from kesselb January 11, 2023 16:29
Copy link
Member

@PVince81 PVince81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 nice one!

@PVince81
Copy link
Member

@Glandos can you rebase ? our bot cannot rebase on forks

this should help bring the branch up to date and help wake up the CI jobs that got stuck

@Glandos
Copy link
Contributor Author

Glandos commented Jan 14, 2023

@PVince81 commented on Jan 11, 2023, 9:01 PM GMT+1:

@Glandos can you rebase ? our bot cannot rebase on forks

this should help bring the branch up to date and help wake up the CI jobs that got stuck

Originally posted by @PVince81 in #35129 (comment)

This is now rebased on current master.

@szaimen szaimen added 4. to release Ready to be released and/or waiting for tests to finish and removed 3. to review Waiting for reviews labels Jan 14, 2023
@juliushaertl
Copy link
Member

Failures seemed unrelated but restarted cypress to be sure ;)

@PVince81
Copy link
Member

hmmm, is background rendering using the previews endpoint ?

 /home/runner/work/server/server/cypress/snapshots/actual/theming/user-background     (1280x720)
     .cy.ts/User default background settings -- Default cloud background is not rende               
     red (failed).png

@blizzz blizzz mentioned this pull request Feb 1, 2023
@skjnldsv skjnldsv mentioned this pull request Feb 23, 2023
@blizzz blizzz mentioned this pull request Mar 7, 2023
@blizzz blizzz removed this from the Nextcloud 26 milestone Mar 9, 2023
@blizzz blizzz added this to the Nextcloud 27 milestone Mar 9, 2023
getCachedPreview used to call `getFile`, and this calls `getDirectoryListing` (or underlying function that list directory) to find the file. This was done for every preview spec.
Now, this is done only once at the beginning of the loop, and the array is just iterated when needed to find the correct entry.

Signed-off-by: Glandos <bugs-github@antipoul.fr>
Signed-off-by: Glandos <bugs-github@antipoul.fr>
Signed-off-by: Glandos <bugs-github@antipoul.fr>
Signed-off-by: Glandos <bugs-github@antipoul.fr>
Signed-off-by: Glandos <bugs-github@antipoul.fr>
getDirectoryListing should return all files
getFile is not called anymore

Signed-off-by: Glandos <bugs-github@antipoul.fr>
use directory listing in both functions to gain 25% speed on run where
every preview already exist.

Signed-off-by: Glandos <bugs-github@antipoul.fr>
Signed-off-by: Glandos <bugs-github@antipoul.fr>
@szaimen szaimen enabled auto-merge April 17, 2023 14:29
@szaimen szaimen merged commit 5cda8f0 into nextcloud:master Apr 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
4. to release Ready to be released and/or waiting for tests to finish feature: previews and thumbnails performance 🚀
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants