How many files/folders can Allusion handle?

[hummingly]

Discussed in #421

^{Originally posted by cptvincer February 5, 2022}
I know thats very vague and would be highly dependent on other factors such as hardware and that its subject to change between builds- but ive been trying to figure out by myself and ive been wondering how its been for other users.

Im also having a hard time figuring out how things work in Allusion. Ive tested adding different folders/subfolders, on my internal drive, from a external one, little by little or many at once and i got different results:

Test-Huge Import: i added a root folder from a external hard drive that probably had around 300k (if not more) files between various folders and degrees of nesting.
It took awhile... but to my surprise it worked(!). When it finished Allusion was consuming a lot of ram (5gb+) and the file count was lost (like i mentioned in a open bug issue), showing as 0- but querying for folders or tags was working.
Crash: It only happened when i restarted Allusion- during the 'checking for new images' process, high cpu, ram and disk usages and taking awhile until the app crashed

40k multi-nested import: i then tried importing a subfolder with much less files from the ones i did the first test before.
This time the total file count wasnt lost and showed something around 40k files.
Crash: but that setup crashed again on 'checking for new files' just like the prior test.

90k+ pictures, no crash: i realized subfolders could be re-included and did another test, starting with 1 folder full of subfolders (1 nesting level only) but only including a few of the folders; Later i was re-including and restarting the remaining folders little by little and all worked fine.
So i kept going- adding folders that had 1 level of subfolders (most of then, a few had 1 other subfolder level).
At one point the library reached 90k+ files, restarting and surviving the 'checking for new files' just fine.
Lost count file: i kept pushing little by little. Around the time the library was showing around 90k file count i added (re-included) more folders (some i didnt realize were actually big), and upon restarting the file count was lost.

this is where i am right now, not sure if i keep testing or if i figure out how to better use Allusion around this not-precise '90k' mark, rotating file folders as needed (ex: no need for animals and nature references if im looking for portraits). Doing that is doable albeit cumbersome, the import/export tags as metadata the silver lining that would allow me to not loose tags while swapping folders that way.
But of course i would prefer not swapping things around and filtering via tags across everything. Hopefully in the future with isolated portable version (db inside portable folder) i could have everything in Allusion even with limits, making some separated libraries

My Impressions so far:

• checking for file changes across everything on startup seems to be the main limit. I cant tell how many files to how much performance Allusion could actually hold without that
• Folder/Subfolder count and or nesting (or length of file paths) seem to affect things; The bug of loosing all file count and the crash on checking files seem to hit much earlier the more folders or nesting is involved
• Ram usage is higher after import then normal use: at one point i was afraid of the proportion of file count to performance was too high, but upon restarting the picture was completely different. Im currently at 90-100k files with very low ram usage.

I think that an option to disable check-for-new files on startup, option to manually check per folder and a option to turn off how Allusion can identify external changes imediatly would tremendously increase how much Allusion can handle, also would give options to those with lower hardware specs.

Hows your libraries going? How many files/folders?

Related: #384

[hummingly]

With the upcoming saved search feature we should look at ways to improve performance as there will be many cases where users will make relatively few queries but have many files.

[cptvincer]

Adding to that original post, i got to 382.192 files before i decided it wasnt worth pressing the stress testing any further (and at that point it was getting out of even my scope). I coulve pushed further but the ram usage on startup was the roadblock, reaching 6-7gb and taking a long while 'checking for new files'.

Ive kept the file dump i did to keep testing, even tough thats more then i would be actually be tagging right now just to see how things goes... second day now, the number is certainly lower a few thousand as ive been cleaning up a lot of files within Allusion (cant tell how much, with so many files allusion doesnt show the total count) and it runs smoothly at 900-1900 ram at max when selecting or shifting to larger folders.

Its just the startup all directory scan that throtles the app, and it take over an hour or more for the ram usage to lower (it slowly ramps down after startup, i cant tell if the pace changes, if other apps asking for ram affect it, etc). I feel i havent really stress tested Allusion yet, i wonder how much it would take under normal usage/load (without the initial huge spike).

But so far said normal usage when things wind down seen pretty great for me.

[PudiccaA]

Adding a few points:

1. Just curious if anyone has tested Allusion on SSD? Currently I have my Allusion library all on HDD and it takes a while to load up images after a search operation. I am planning to move all pictures to a SSD and see if it helps.
2. When starting up the application, the default manner now is load up ALL pictures in the masonry, which gave a performance hit. I'd personally prefer an option to start up the application with a blank gallery, such as the one returned if no pictures found upon doing a search.

---

Did a few experiments and I can confirm that moving images to SSD didn't help improve the image loading time at all.

[cptvincer]

Adding a few points:

1. Just curious if anyone has tested Allusion on SSD? Currently I have my Allusion library all on HDD and it takes a while to load up images after a search operation. I am planning to move all pictures to a SSD and see if it helps.
2. When starting up the application, the default manner now is load up ALL pictures in the masonry, which gave a performance hit. I'd personally prefer an option to start up the application with a blank gallery, such as the one returned if no pictures found upon doing a search.

Did a few experiments and I can confirm that moving images to SSD didn't help improve the image loading time at all.

I wanted to test that too (but my only sdd is full of applications and some caches i cant live without)- im glad someone did! It confirms/re-inforces the 'startup file check' as the bottleneck. The SSD probably would make a difference if it was all about file loading or scanning alone, but the way Allusion seens to work it needs to load a lot on memory (and probably should, otherwise it wouldnt be as snappy when loaded).

My guess is memory will always be the biggest factor, at least in regards to loading a bigger folder or doing a scan (if/when theres a option to do it manually). BUT i imagine theres room for vast improvement if some future build bypass (or gives us the option) the 'whole directory scan' on startup, or the realtime update to all folders (in regards to external changes).
Theres some buildup on memory/trash collection the more there is to scan at once- so my theory is that doing that by demand (even without optimizing the memory buildup) could make a really nice difference just by making the workload partial, during use.

[Jopp-gh]

Feature-wise, everything works greatly, Allusion is amazing

The only thing that needs improvement, Allusion cpu load is constantly over 11% - 15%, this is definitely not a normal value
Apps in idle shouldn't consume more than cpu loads of 0.x %

Cpu loads return to normal values immediately after quitting Allusion

[RvanderLaan]

Hi @johans3, that's strange, I have not noticed that myself. Not sure if it's related to this issue though.

If you want to help solve this, could you create a new issue for this? It would help if you could:

• Mention what OS you're running
• Check if it's a specific (sub)process or the main process of Allusion that is doing the harm
• Check if there are any weird/repeating logs in the Developer Tools console (see the Advanced tab in the settings panel to open this)