dirty python packages in dist

[kaos]

When running ./pants package :: any previous packages in ./dist/ ought to be removed first in order to clean out old files for any target packages.

I have noticed that I have to manually remove them in case I re-organize my code, the now no longer referenced files are otherwise still left in the tree.

[Eric-Arellano]

Would a warning have helped perhaps? I suspect there are cases where you want to run ./pants package multiple times sequentially without Pants erasing the prior runs.

[kaos]

A warning would be nice, and maybe even a ./pants --clean ... to opt-in to it ;)

[kaos]

Just thought to clarify, that if you run seq with different targets, it wouldn't be an issue, as I suggest to only remove the target that is about to be written. So any seq run ought to produce the same result, everything else unchanged. Not sure if that came across in my previous post.

[Eric-Arellano]

Oh! I did not realize that's what you were proposing. What are you creating here? I think we hadn't considered this as being necessary because package originally only created single files, like .pex files, which got overwritten. But I realize python_distribution can create a loose directory.

[kaos]

I use that to get a virtualenv where I can install several (all) my dists in to play around and test with. Couldn't find there is a better/recommended way of doing this..?

EDIT:
What I do is, I run package to get my dists into, well, the dists dir. From there, I run python setup.py develop for each dist (from my own virtualenv activated)
That way, I can use any console scripts etc, and it works as it would if installed from pypi. Also, to update, I simply run ./pants package again, and since pip points to my dists tree for the sources, I don't have to do anything else to get my env updated.

EDIT2:
Snippet from my Makefile to setup a virtualenv..

dev:
	./pants package ::
	for d in `find dist -name setup.py`; do \
		pushd $${d%/*} ; python setup.py develop ; popd ; \
	done

[jsirois]

I use that to get a virtualenv where I can install several (all) my dists in to play around and test with. Couldn't find there is a better/recommended way of doing this..?

It depends on what sort of playing around you do. If its experimenting in a repl, then ./pants repl :: directly addresses the need. If its playing around in some other tool that has venv support, we currently fall a bit short. The ability to create virtualenvs (to support IDEs) was the top vote getter in the Fall 2020 community prioritization effort. I think that's most closely tracked by #11152. Today we can only offer this for pex_binary targets individually:

1. Either add include_tools=True or execution_mode="venv" to the pex_binary target.
2. Run ./pants package my/pex:binary
3. Run PEX_TOOLS=1 dist/my.pex/binary.pex venv create/venv/here (Add --pip if you want pip installed in the venv to mutate with)
4. Run source create/venv/here/activate

[kaos]

Yeah, the repl doesn't really offer the cli entry point experience I'm after here.
That's a neat trick for pex'es. I wonder, though, if I ran package again, are the changes reflected in the created env, or do I need to recreate it.. ?
As I do now, with using the packages from dist directly, I don't need to do anything after running package to get the updates into my venv, which is a big win on turn-around times.. (best would be if the sources were symlinked into the source tree, then I wouldn't need to repackage it even, haha)

[jsirois]

That's a neat trick for pex'es. I wonder, though, if I ran package again, are the changes reflected in the created env, or do I need to recreate it.. ?

Yeah - you'd need to re-create it. The develop mode trick is most closely matched by ./pants run my/pex:binary. To run arbitrary entry points, whether they be in your code or third party code, use PEX_MODULE or PEX_SCRIPT. For example, in the Pants repo, where the pants pex_binary depends on mypy:

$ PEX_SCRIPT=mypy ./pants run src/python/pants/bin:pants -- --version
mypy 0.800

[kaos]

Cool, will have to give that a spin, see how it goes :)

EDIT: Nice, that runs it cool, only one thing missing, and that is that the pex file doesn't include my dists as wheels, but as source only.. so I can't dynamically load them using setuptools entry_points.. :/

[ghost]

Not sure this is the right issue, but this discussion has been the closest to my problem: I have a mono repository setup with VSCode and many poetry projects. We used to create a venv with all locked dependencies and the packages in the repository as editable installs. The nice effect of this was that VSCode then does auto completion, but also debugging and test runs are easy to configure and changed code is always reflected.

To me it seems that the debug/development workflow is completely missing from pants, at least when I want to use other tools/IDEs that expect a venv with all the third party dependencies installed. Is there some workaround or is this planned for the near future? I am not sure I fully understand the PEX_MODULE or PEX_SCRIPT hint above.

[stuhood]

Not sure this is the right issue, but this discussion has been the closest to my problem: I have a mono repository setup with VSCode and many poetry projects. We used to create a venv with all locked dependencies and the packages in the repository as editable installs. The nice effect of this was that VSCode then does auto completion, but also debugging and test runs are easy to configure and changed code is always reflected.

Hey @malte-klemm-by ! I think that this issue is mostly unrelated to what you are trying to do: to set up an IDE for a Pants repo, see https://www.pantsbuild.org/docs/setting-up-an-ide. The only way that this issue might affect your IDE setup would be if you are using codegen (generated code): in that case the ./pants export-codegen :: step would need to be re-run.

[kaos]

Unrelated to this issue, yes. However the reason I got here is related. I too was looking for a good dev/debug feedback-loop.

I think we can do better here, will open dedicated ticket if none exist when I get back from vacation.

[ghost]

Hi @stuhood, first of all thanks for the quick answer. I had read that section in the docs, but the setup described there is not really ideal for vscode.

First, adding each root to the vscode settings is something we had to do in the past and didn't work well (we have around 40-50 root folders and making sure these settings work with every developer proved to be quite error prone). Secondly, setting up the remote debugger in VSCode is always a bit more fiddling than using the direct launch.json to start a process and have a debugger attach on breakpoints or exceptions.

My hope was somehow that the PEX_SCRIPT allows me to wrap the vscode debugger around the actual script (that was what triggered my attention). However, it seems that vscode expects to start a python interpreter from a venv for python debugging in the launch.json format and this doesn't work with a pex file as the first entrypoint (unless there is a way to run a pex from an existing python interpreter via -m flag somehow).

I had a look at the plugin documentation and have some ideas how I could get to the expected setup. I will have a look at it after my vacation and will add to the discussion on the new ticket / join the slack to discuss my setup and give some feedback. In our team we have used multiple mono repo approaches so far and the biggest downside has always been the dev/debug setup so I am happy to improve as pants v2 looks really promising apart from that aspect.

[stuhood]

Ah, thank you both for the clarification and investigation!

My hope was somehow that the PEX_SCRIPT allows me to wrap the vscode debugger around the actual script (that was what triggered my attention). However, it seems that vscode expects to start a python interpreter from a venv for python debugging in the launch.json format and this doesn't work with a pex file as the first entrypoint (unless there is a way to run a pex from an existing python interpreter via -m flag somehow).

PEX does support a venv execution mode, and it is enabled by default for most internal PEXes that are built (including those used for tests). One thing that can help with investigating the test sandboxes is to run with --no-process-execution-local-cleanup: it's possible that you could expose support for rendering the relevant venv location to better support attaching a debugger.

[kaos]

Fixed by #18639