poetry build ignores locked versions

[ijanos]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request ?

When running poetry build the wheel metadata seems to be generated based on the pyproject.toml instead of the lock file. Is this intentional? Maybe I am missing something but I thought calling poetry build will create a wheel that will install the exact same dependencies from the lock file.

here's what the dependencies look like:

[tool.poetry.dependencies]
python = "^3.6"
flask = "^1.0"
requests = "^2.21"
redis = "^3.2"
lxml = "^4.3"
pillow = "^6.0"

And here is the relevant part of the METADATA file from the generated wheel. I expected these to be == exact versions.

Requires-Dist: flask (>=1.0,<2.0)
Requires-Dist: lxml (>=4.3,<5.0)
Requires-Dist: pillow (>=6.0,<7.0)
Requires-Dist: redis (>=3.2,<4.0)
Requires-Dist: requests (>=2.21,<3.0)

[mwchase]

From https://poetry.eustace.io/docs/libraries/#lock-file:

For your library, you may commit the poetry.lock file if you want to. This can help your team to always test against the same dependency versions. However, this lock file will not have any effect on other projects that depend on it. It only has an effect on the main project.

I interpreted that as "the lock file only affects installs done from the project itself, as opposed to from built artifacts", which I think is what you're seeing.

[ijanos]

I think you are interpreting the doc a bit too liberally. It talks about that the output of poetry takes the lock file into account but you should not depend on other tools taking the lock file into account when including your project. Which is perfectly fine.

This question/feature request is not related the quoted documentation. I am not asking about nesting my project into another one. I would like the lock file to have an effect on the sdist/wheel build artefacts

[ijanos]

Although since then I realised that the wheel metadata only contains the direct dependencies and not transitive dependencies, unlike the lock file which has every transitive dependency.

I still feel a tighter locking of version numbers should be possible in the wheel metadata, I would love to hear some feedback on this from the developers of the tool.

[dmontagu]

@ijanos can that not be achieved by just including tighter restrictions on the versions listed in the pyproject.toml? E.g., if you want to pin the version of a transitive dependency, just list it in the pyproject.toml with a pinned version?

I think what you are describing might conflict with PEP 518 in some way if it used information outside the pyproject.toml to generate the wheel metadata. (I haven't read it very closely though.)

I'm not sure if there is a way to generate a modified pyproject.toml with the locked requirements (similar to the preview version's ability to export requirements.txt), but even if this doesn't currently exist it would probably be pretty easy to implement by parsing the poetry.lock and using the toml package. That's probably the best bet for distributing a wheel with tightly pinned dependencies. But note that this is more likely to lead to dependency hell if the project using the tightly-pinned wheel has other dependencies that require versions different from those in the tightly-pinned wheel (but which could still safely be upgraded).

Beyond that, it sounds like what you are interested in is something akin to building a wheel with "static linking" to dependencies (i.e., different packages could depend on different versions of the same dependency in the same environment), but I'm not sure whether this is even theoretically possible in python (currently). (Maybe someone more knowledgeable can shed some light..)

---

If you want to distribute sub-projects as wheels without heavy pinning restrictions on the execution environment (so they can be more easily reused), perhaps a better option for any project in which you want to include dependency wheels and still have tight version pinning would be to make use of a poetry "wrapper" project. This project would include all dependencies in its pyproject.toml, would use poetry.lock, and would use poetry install [--no-dev] as the way to construct the execution environment. This could get you the tight pinning you want that might be very frustrating to work with if built directly into any reusable wheels.

[brycedrennan]

Echoing what @dmontagu says,

There are libraries and applications.

A wheel is intended to distribute a library, which generally supports a range of versions for its dependencies. If each library specified strict version dependencies then it would be very hard for two libraries to coexist. Its generally considered bad practice to have tight dependencies for libraries. poetry build is for creating wheels for libraries. These wheels then end up on repositories like pypi.

For applications (which use a libraries as dependencies) its best practice to lock down the dependencies and sub-dependencies so that your application works exactly as expected. This is what lock files are for. Applications are not distributed via wheels. Docker images are a widely used mechanism for distributing applications. You may also have interest in cutting edge things like PyOxidizer.

If for whatever reason you still want really tight dependencies in your wheel then yes you can manually specify them all in your pyproject.toml

[ijanos]

Yeah, I see. I am using a wheel to deploy an application basically like this:

1. use CI to run tests and lints
2. if all OK then run poetry build
3. copy wheel to server
4. create new virtualenv
5. pip install wheel in new virtualenv
6. if everything is fine then switch from old venv to new

I am quite fond of this workflow, my only nitpick is the tighter control over versions but as you suggested I think I will move to tighter specs in the the pyproject file. Docker images and pyoxidizer are both way too heavy for my use case.

Can you back up your claim of wheels are intended for libraries? I was not aware of such distinction and I did look at the PEP but found no such intention it has the following defition:

Wheel provides a Python-specific, relocatable package format that allows people to install software more quickly and predictably than re-building from source each time.

[Tobotimus]

Applications are not distributed via wheels.

Sorry to be necromancing this issue a year later, but this is simply not the case. If wheels weren't intended for distributing apps, console-script entry points wouldn't exist (or at least, would not be so ubiquitous). You might personally prefer to distribute your application as a Docker image rather than a wheel, but not everyone can or should conform to that opinion. A wheel is a perfectly valid distribution for hundreds of applications on PyPI - often it simply involves the user making a virtualenv, running pip install application, and running it with a command. Users in the community of a project I maintained (Red-DiscordBot) have been happily doing this for years. This could even make building application Docker images easier (poetry no longer being a pre-requirement for installing your app as a distribution).

In this use case, it is more suitable for the application's wheel to have all its primary and transitive dependencies pinned to known working versions. An amazing feature for Poetry which would allow its swift adoption as a dependency management tool for applications would be:

poetry build --format=wheel --dependencies=locked

Perhaps it should also/instead be a key in the [tool.poetry] table so the same behaviour is observed whether the project be installed from an sdist, wheel, VCS etc. I know this would be super useful for a number of projects I've been involved with, both in OSS and at work.

[danmur]

Having the option would be really good.

[strangemonad]

+1 to what @Tobotimus said. This sort of workflow goes hand in hand with tools like pipx which install apps from wheels and with entry-points.

[garawalid]

+1 to @Tobotimus proposal.

[tardyp]

new issue about this is #2778

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.