Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Smarter Backoffs for MPI ring connection #2348

Closed
mhamilton723 opened this issue Aug 22, 2019 · 2 comments · Fixed by #2354
Closed

Add Smarter Backoffs for MPI ring connection #2348

mhamilton723 opened this issue Aug 22, 2019 · 2 comments · Fixed by #2354

Comments

@mhamilton723
Copy link
Contributor

mhamilton723 commented Aug 22, 2019

Hey, I am running many LightGBM training runs as part of LightGBM on Spark's CI process and i notice that it does alot of waiting because of bad network and this dramatically slows the tests:

const int connect_fail_delay_time = 10 * 1000; // 10s

Could you potentially make it first wait for half a second, then wait for 10 seconds? Or make this configurable. Thanks for the help!

@guolinke
Copy link
Collaborator

This could be configurable.
I will create a PR for it.

@mhamilton723
Copy link
Contributor Author

Thank you @guolinke for considering this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants