-
Notifications
You must be signed in to change notification settings - Fork 981
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
throttle threads for iterated small data tasks #4484
Conversation
Codecov Report
@@ Coverage Diff @@
## master #4484 +/- ##
=======================================
Coverage 99.61% 99.61%
=======================================
Files 73 73
Lines 14103 14108 +5
=======================================
+ Hits 14048 14053 +5
Misses 55 55
Continue to review full report at Codecov.
|
Related: #3205 (comment) |
This comment has been minimized.
This comment has been minimized.
I just couldn't grasp the types so I changed this PR to what clicks in my head, at least. See what you think. The names OMP_ROWS and OMP_BATCH didn't click because in some cases that are not directly to do with rows, you do want to throttle. I just find it easier to see a bool flag (true/false) telling me directly whether the throttle should be applied or not in that region, so I don't have the mental hop via the type codes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice! much simpler approach than mine
@mattdowle I updated timings in first post. It is interesting to see that |
Just to ensure this PR addresses the problem on Windows platform as well. before patch
after patch
Note that issue #4527 is about single threaded overhead, which on windows seems to be much bigger than on linux. Here problem is the overhead caused by many threads. |
Closes #3175, see #3175 (comment)
Closes #3205 (the 2nd part which was outstanding)
Closes #3438: rerun repeated uniqueN test
before patch
after patch
partially address #3735:
output before patch
after patch
partially address "slow uniqueN" #3739
before patch
after patch
slow uniqueN by group #1120
before patch
after patch