Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimizations #48

Merged
merged 11 commits into from
May 31, 2021
Merged

Optimizations #48

merged 11 commits into from
May 31, 2021

Conversation

coreprocess
Copy link
Contributor

@coreprocess coreprocess commented May 24, 2021

TL;DR: The PR reduces 1000x calls of load in a loop from a total time of 1.7s to 0.01s (within a VMWare VM).

  1. We removed the thread pool and are using worker queues instead.
  2. Default file type detection is based on extensions now. Magic based detection can still be activated.

Note: The threads and the timer are registered on demand and will be removed automatically when all items have been processed.

coreprocess and others added 7 commits May 23, 2021 17:48
And use pathlib to get the suffix.
I think this approach is cleaner.
This allows the code after it to be tabbed back, which I prefer.
Also I changed comments a little.
I think it's good to have this documented.
This makes it more obvious what it's for.
Adjusted some comments, newlines, formatting.
We want to catch it if a queue item is somehow not a correct size tuple, so I'm just doing a generic except instead of excepting Empty.
Variables that end in `sig` now end in `_signal`.
`_read_thread` no longer takes `thread_index`, as it wasn't using it.
Swapped the order of the signal check and try block.
Removed manual unregister of timer, as returning None is enough.
There are multiple threads.
All the other files follow a plural naming scheme.
bonjorno7
bonjorno7 previously approved these changes May 31, 2021
Copy link
Owner

@bonjorno7 bonjorno7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with the state of this branch.
Did make some of my own changes, so you could take a look at those if you want.

@coreprocess coreprocess merged commit 5dc9d3c into main May 31, 2021
@coreprocess coreprocess deleted the feature/optimizations branch May 31, 2021 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants