-
Notifications
You must be signed in to change notification settings - Fork 298
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implements variably-sized caches in 'RasterDataset' #1695
base: main
Are you sure you want to change the base?
Conversation
572676b
to
999f06d
Compare
@microsoft-github-policy-service agree |
The current error on
|
Hey @pmaldonado, we're aware that the windows test is failing for unrelated reasons. Going to give it a day to see if it resolves, then look into it. |
I wonder if we could somehow detect the total RAM size and set the default cache size based on that. I don't really want to add a new dependency on psutil though. |
Should "GeoDataset" change it's behavior based on properties of the DataModule (the number of workers) or system (amount of RAM)? A lightweight solution to our issue is allow users to overwrite the An alternate implementation could be allow the |
999f06d
to
711e1a2
Compare
I think the filename LRU cache is shared across all processes so it isn't affected by the number of workers, but correct me if I'm wrong. That's just an assumption, I have no evidence.
This depends. If we know that one dataset uses COGs and one doesn't, that may affect how large we want the cache to be. But let's explore |
This is fine with me! |
@pmaldonado have you tried |
Ping @pmaldonado |
Thanks for the ping! Got pulled into a few other things. I'll give your suggestion with |
Closes #1694.