Prepare to enable pantsd by default #9826
Conversation
stuhood
force-pushed
the
stuhood/pantsd-by-default-questionmark
branch
2 times, most recently
from
e4c3d85
to
39a4e6e
Compare
# Delete this line to force CI to run Clippy and the Rust tests. [ci skip-rust-tests] # Delete this line to force CI to run the JVM tests. [ci skip-jvm-tests]
stuhood
force-pushed
the
stuhood/pantsd-by-default-questionmark
branch
2 times, most recently
from
2bcc5f4
to
875e4f7
Compare
[ci skip-rust-tests] [ci skip-jvm-tests]
…hem being gitignored. # Delete this line to force CI to run the JVM tests. [ci skip-jvm-tests] # Delete this line to force CI to run Clippy and the Rust tests. [ci skip-rust-tests]
stuhood
force-pushed
the
stuhood/pantsd-by-default-questionmark
branch
from
875e4f7
to
da95bb2
Compare
| @@ -134,11 +134,18 @@ REQUIREMENTS_3RDPARTY_FILES=( | |||
| # internal backend packages and their dependencies which it doesn't have, | |||
| # and it'll fail. To solve that problem, we load the internal backend package | |||
| # dependencies into the pantsbuild.pants venv. | |||
| # | |||
| # TODO: Starting and stopping pantsd repeatedly here works fine, but because the | |||
| # created venvs are located within the buildroot, pantsd will fingerprint them on | |||
There was a problem hiding this comment.
We do encourage users to create venvs within the build root: https://pants.readme.io/docs/python-third-party-dependencies#tip-set-up-a-virtual-environment-optional
Presumably, those will be in their .gitignore already, so it would be in pants_ignore. Should this be okay for end users?
There was a problem hiding this comment.
This is referring to a virtualenv that contains pants' own code, rather than to end user virtualenvs. pantsd fingerprints its own sys.path and source plugins in order to restart if they are edited.
| ): | ||
| """Temporarily create the given literal directory, which must not exist.""" | ||
| path = os.path.realpath(path) | ||
| assert path.startswith( |
There was a problem hiding this comment.
This assert is surprising given the doc. Seems worth documenting or else moving the caring about this to the call site that has this restriction.
|
|
||
| parent = os.path.dirname(path) | ||
| parent_ctx = ( | ||
| suppress() if os.path.isdir(parent) else self.temporary_directory_literal(parent) |
There was a problem hiding this comment.
I think you can use os.makedirs to get rid of the recursion and existence checks: https://docs.python.org/3/library/os.html#os.makedirs
There was a problem hiding this comment.
The recursion is to try to leave things as we found them: we recurse upward to create the parents temporarily, rather than permanently as with makedirs. Expanded the method docs.
…links, and is no longer strictly necessary for isolation when run with v2. [ci skip-rust-tests] [ci skip-jvm-tests]
…tead, 2) enable only for the pantsbuild/pants repo, 3) test helper cleanup. [ci skip-rust-tests] [ci skip-jvm-tests]
stuhood
force-pushed
the
stuhood/pantsd-by-default-questionmark
branch
from
298682a
to
307af2e
Compare
Problem
Pants' runtime overhead is influenced by multiple factors:
@rules with unchanged inputsPantsd helps to address all of these issues, and has just (re-)reached maturity.
Solution
Prepare to enable
pantsdby default by deprecating not setting theenable_pantsdflag. Also, set the flag for pantsbuild/pants.Result
noop time when running in a virtualenv (such as in github.com/pantsbuild/example-python) drops from
~2.4sto~1.6s. Followup work (moving fingerprinting to the server, porting the client to rust) is expected to push this below500ms.There are a collection of known issues that might impact users tracked in https://github.com/pantsbuild/pants/projects/5, some of which we will address via dogfooding. Others are matters of expectation: "I wouldn't expect that to work".
One of the most significant issues though is #7654: we should consider making a push before
1.29.0to remove use of global mutable state to allow for concurrent pants runs with pantsd under v2.Fixes #4438.