What does it actually mean to install a git dependency in editable mode?

[ckp95]

Hi, I see in the documentation it's possible to install a git dependency in editable mode:

poetry add --editable git+https://github.com/some-repository

But I'm having trouble understanding what this actually means. How do I actually .. you know, edit it? Where does it go? I don't see any repo created in my project directory when I do this, although I am able to import it. What is the expected workflow, what am I meant to do with it?

[neersighted]

This would probably be a good FAQ item, or maybe something to add around the dependency management section.

To answer your question, this is a bit of a 'completeness' feature for parity with pip. I wouldn't consider it super useful for most Poetry users, but it was requested and it was easy to implement, plus some external tools might want to make use of it.

When you install a VCS url in editable mode, it hybridizes the install modes. If I fully explain it, this will probably make sense:

• When you install a VCS dependency, it is cloned into $VENV_ROOT/src/<name> where name is the #egg= name (often determined from setup.py egg_info or a PEP 517 build).
• When you install a dependency in editable mode, instead of 'building' it (having the build system copy all 'installable' files into a path/archive and generate metadata, then copy that into site-packages), a .pth marker is added. If you inspect $VENV_ROOT/lib/python3.x/site-packages you'll see a .pth entry for your 'root package' (aka your Poetry-managed project).

So when you install a VCS dependency in editable mode, the repo is cloned into the $VENV_ROOT/src folder, and a .pth marker is created to use that VCS repo as the source of imported Python code (as opposed to the usual process of a build-time snapshot and copy of the Python files).

I hope that helps -- a MR cleaning up my explanation and adding it to the docs would be very welcome!

[ckp95]

Ah okay that's actually pretty neat, I could make use of that. I sometimes have to simultaneously develop on a "upstream" and "downstream" project at the same time, where the downstream project depends on the upstream project's git repo. So being able to edit the upstream and see the changes straightaway in my local dev version of downstream is nice. I don't have to constantly push and pull the changes.

It is a little awkward to have the repo inside the virtual environment though ... I just tried symlinking it into the root of the project folder for convenience and it seems to work, although it feels a little iffy. Would there be any problems caused by doing this? i.e.

cd downstream-project
ln -s .venv/src/upstream-project/ ./upstream-project

It seems to import the same way.

[neersighted]

So by spec, VCS deps have to be present at $VENV_ROOT/src/<name>... But there are no other assumptions made around them, as far as I am aware.

That being said, be cautious of leaning on this too much, as I don't think it was ever intended for longer-term use -- poetry install should respect your repo, but poetry update may check out files from a different ref in my experience. Basically I would commit often and make use of the reflog as I'm not 100% sure the behavior around updating is fully defined for editable VCS deps.

[ckp95]

Yes I just tested it, poetry update will clobber any local commits made, without telling you. It snaps the repo back to whatever the remote origin is, although new files added to the working directory persist but in uncommitted state ... weird.

Am I right in thinking a path dependency would be safer?

[neersighted]

I think technically what happens is poetry update will git checkout <ref tracked in lockfile> <paths to packages declared in pyproject toml> instead of changing the checked out ref entirely when the ref is mismatched -- I don't remember why this is, but I believe it's to prevent breaking your project completely by switching to a branch with a different layout or similar.

An editable path dependency is the classical solution, yes (putting the dependency back when you are done). One day we might grow support for local replacements or similar, but for now there's not a great way to do that.

Okay, the 1.2 betas did allow for one to have multiple copies of a dep, in different groups, with different source URLs. However that was actually the result of a bug that caused some other problems. With the current architecture, two deps that are identical except in terms of source URL are considered the same and will collide/cause your project to become uninstallable.

[ckp95]

Okay, that makes sense. Oh one other thing, maybe this is a bug: in pyproject.tomlI changed my dependency from the editable git repo to a editable path, and ran poetry update. However the .venv/src/upstream-project directory persisted instead of being deleted. Is there a way to make it scrub those files so they don't cause confusion? i.e. if I had a symlink pointing to that repo, I would want that symlink to break, rather than allow me to make useless edits via it by mistake.

[neersighted]

I think we don't currently clean up as we use pip to manage installations, and pip will just install the new package when you switch back (deleting/creating .pth, copying source files into site-packages and writing the .dist-info as necessary).

It is possible #6205 could change this as then the installer would become part of Poetry, and aware of more than the immediate task (install xyz package). That being said, it does look like poetry install --sync gets confused by the presence of an orphan Git repo in the virtual environment -- it's possible that that command could be updated to identify this situation and nuke the repo (it's stated purpose is to 'make the venv exactly match the lockfile, nuking anything that doesn't have express permission to be there').

[lainisourgod]

My current setup is that I have a

• common package that is used by
  -service_a, service_b, service_c

These four are different repos, but common can't be used outside of one of these services. So I usually develop common logic only in conjunction with service_x logic.

In this way, I need common package to be both editable install and a git dependency, so I can both interactively see changes in common from service_x without need to do constant push-pulls, but also able to commit the changes in common as I make them.

Don't really know how to do this in current poetry workflow.

[bersbersbers]

@lainisourgod I was thinking about making common a Git submodule in all service_*, and then doing an editable install from that submodule folder. So there is an implicit git dependency if you will. (It would mean you have three working copies of common, though.) I haven't tried this yet as I would have to change to poetry for that (but I would) - have you tried this by any chance?