Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase default [python-setup].resolver_jobs to cpu_count / 2 #11006

Merged
merged 1 commit into from
Oct 21, 2020
Merged

Increase default [python-setup].resolver_jobs to cpu_count / 2 #11006

merged 1 commit into from
Oct 21, 2020

Conversation

Eric-Arellano
Copy link
Contributor

@Eric-Arellano Eric-Arellano commented Oct 21, 2020 •
edited
Loading

This option controls the concurrency of each Pex subprocess.

We had set the default low to 2 because—before we added the single resolve optimization—it was common to build multiple PEXes at the same time. This resulted in exhaustion / an OOM issue for Toolchain, which was tricky to figure out.

Now, the user normally builds no more than 1-3 PEXes at a time, unless running ./pants package :: or does not use a constraints file so we can't use a single resolve. So, our default is missing out on performance.

PEX's normal default is cpu_count. Instead, we use cpu_count / 2 to try to balance performance vs. safety.

[ci skip-rust]
[ci skip-build-wheels]

@Eric-Arellano
Increase default [python-setup].resolver_jobs to cpu_count / 2
6d9ca5f 
# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
Eric-Arellano
Eric-Arellano commented Oct 21, 2020
View reviewed changes
Copy link
Contributor Author

@Eric-Arellano Eric-Arellano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try to balance performance vs. safety.

I think this change is safe enough, but am still scarred from the OOM issue.

src/python/pants/python/python_setup.py
advanced=True,
help=(
"The maximum number of concurrent jobs to build wheels with. Because Pants "
"can run multiple subprocesses in parallel, the total parallelism will be "
"can run multiple subprocesses in parallel, the maximum total parallelism will be "
"`--process-execution-{local,remote}-parallelism x --python-setup-resolver-jobs`. "
"Setting this option higher may result in better parallelism, but, if set too "
"high, may result in starvation and Out of Memory errors."
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It didn't seem worth it to mention the edge cases where you want this low vs. high. Way too much detail.

@coveralls
Copy link

Coverage Status

Coverage remained the same at 0.0% when pulling 6d9ca5f on Eric-Arellano:jobs into 56b2562 on pantsbuild:master.

tdyas
tdyas approved these changes Oct 21, 2020
View reviewed changes
Copy link
Contributor

@tdyas tdyas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Eric-Arellano Eric-Arellano merged commit 4fd746c into pantsbuild:master Oct 21, 2020
@Eric-Arellano Eric-Arellano deleted the jobs branch October 21, 2020 20:44
Eric-Arellano added a commit to Eric-Arellano/pants that referenced this pull request Oct 21, 2020
@Eric-Arellano
Increase default [python-setup].resolver_jobs to cpu_count / 2 (p…
838c292 
…antsbuild#11006)

This option controls the concurrency of each Pex subprocess. 

We had set the default low to 2 because—before we added the single resolve optimization—it was common to build multiple PEXes at the same time. This resulted in exhaustion / an OOM issue for Toolchain, which was tricky to figure out.

Now, the user normally builds no more than 1-3 PEXes at a time, unless running `./pants package ::` or does not use a constraints file so we can't use a single resolve. So, our default is missing out on performance.

PEX's normal default is `cpu_count`. Instead, we use `cpu_count / 2` to try to balance performance vs. safety.

[ci skip-rust]
[ci skip-build-wheels]
Eric-Arellano added a commit that referenced this pull request Oct 21, 2020
@Eric-Arellano
Increase default [python-setup].resolver_jobs to cpu_count / 2 (#…
441c58b 
…11006) (#11011)

[ci skip-rust]
[ci skip-build-wheels]
@Eric-Arellano Eric-Arellano mentioned this pull request Oct 30, 2020
Closed
@gshuflin gshuflin mentioned this pull request Oct 30, 2020
Merged
gshuflin added a commit that referenced this pull request Oct 31, 2020
@gshuflin
Prepare 2.1.0.dev0 (#11089)
bf5e789 
Internal-only changes: 

* upgrade to cpython crate v0.5.1 (#11052)
  `PR #11052 <https://github.com/pantsbuild/pants/pull/11052>`_

* Prepare 2.0.0 (#11053)
  `PR #11053 <https://github.com/pantsbuild/pants/pull/11053>`_

* Revert "Add new EngineAware method metadata() (#11030)" (#11047)
  `PR #11030 <https://github.com/pantsbuild/pants/pull/11030>`_
  `PR #11047 <https://github.com/pantsbuild/pants/pull/11047>`_

* Remove deprecated `python_binary` target in favor of `pex_binary` (#11046)
  `PR #11046 <https://github.com/pantsbuild/pants/pull/11046>`_

* Prepare 2.0.0rc3 (#11044)
  `PR #11044 <https://github.com/pantsbuild/pants/pull/11044>`_

* Eagerly validate entry points for `setup_py().with_binaries()` (#11034)
  `PR #11034 <https://github.com/pantsbuild/pants/pull/11034>`_
  `PR #11021 <https://github.com/pantsbuild/pants/pull/11021>`_

* Use Ubuntu Bionic for CI (#11027)
  `PR #11027 <https://github.com/pantsbuild/pants/pull/11027>`_

* Prepare 2.0.0rc2 (#11017)
  `PR #11017 <https://github.com/pantsbuild/pants/pull/11017>`_

* Remove RunTrackerLogger (#11018)
  `PR #11018 <https://github.com/pantsbuild/pants/pull/11018>`_

* Upgrade Pex to 2.1.20 (#11014)
  `PR #11014 <https://github.com/pantsbuild/pants/pull/11014>`_

* Remove more unused code from RunTracker (#11012)
  `PR #11012 <https://github.com/pantsbuild/pants/pull/11012>`_

* Add type annotations to AggregatedTimings (#11009)
  `PR #11009 <https://github.com/pantsbuild/pants/pull/11009>`_

* Increase default `[python-setup].resolver_jobs` to `cpu_count / 2` (#11006)
  `PR #11006 <https://github.com/pantsbuild/pants/pull/11006>`_

* Include `<PYENV>` in `[python-setup].interpreter_search_paths` default (#10998)
  `PR #10998 <https://github.com/pantsbuild/pants/pull/10998>`_

* Remove PantsDaemonStats class wrapper (#11003)
  `PR #11003 <https://github.com/pantsbuild/pants/pull/11003>`_
  `PR #files#r508861045 <https://github.com/pantsbuild/pants/pull/11000/files#r508861045>`_

* Revert using libCST for dep inference due to performance (#10907) (#11001)
  `PR #10907 <https://github.com/pantsbuild/pants/pull/10907>`_
  `PR #11001 <https://github.com/pantsbuild/pants/pull/11001>`_


* Run tracker refactor (#11000)
  `PR #11000 <https://github.com/pantsbuild/pants/pull/11000>`_

* refactor Core::new including command runner setup (#10993)
  `PR #10993 <https://github.com/pantsbuild/pants/pull/10993>`_
  `PR #10960 <https://github.com/pantsbuild/pants/pull/10960>`_

* Allow changing the versioning scheme for `python_distribution` first-party dependencies (#10977)
  `PR #10977 <https://github.com/pantsbuild/pants/pull/10977>`_

* Remove tokio Handle type from Executor::new (#10980)
  `PR #10980 <https://github.com/pantsbuild/pants/pull/10980>`_

* Remove deprecated `Address.parse()` and `Address.reference()` (#10981)
  `PR #10981 <https://github.com/pantsbuild/pants/pull/10981>`_

* Remove project_ methods from the externs module (#10955)
  `PR #10955 <https://github.com/pantsbuild/pants/pull/10955>`_

* Prepare 2.0.0rc1 (#10972)
  `PR #10972 <https://github.com/pantsbuild/pants/pull/10972>`_

* Fix interpreter selection when building a PEX to use `[python-setup].interpreter_search_paths` (#10965)
  `PR #10965 <https://github.com/pantsbuild/pants/pull/10965>`_

* Delete `JsonReporter` (#10964)
  `PR #10964 <https://github.com/pantsbuild/pants/pull/10964>`_

* Fix log (#10959)
  `PR #10959 <https://github.com/pantsbuild/pants/pull/10959>`_

* Expose getattr method for Python-related APIs (#10953)
  `PR #10953 <https://github.com/pantsbuild/pants/pull/10953>`_

* Upgrade tokio package to 0.2.22 (#10949)
  `PR #10949 <https://github.com/pantsbuild/pants/pull/10949>`_

* Use `package` to build Pants's wheels, rather than `setup-py` (#10947)
  `PR #10947 <https://github.com/pantsbuild/pants/pull/10947>`_

* Simplify val_to_str and have it and val_to_log_level use PyObject (#10946)
  `PR #10946 <https://github.com/pantsbuild/pants/pull/10946>`_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tdyas tdyas tdyas approved these changes

@benjyw benjyw Awaiting requested review from benjyw

@gshuflin gshuflin Awaiting requested review from gshuflin

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Eric-Arellano @coveralls @tdyas