Dependabot update to single (poetry-managed) package unexpectedly removes packages from lockfile

[DMRobertson]

Is there an existing issue for this?

• I have searched the existing issues

Package ecosystem

Python (poetry)

Package manager version

No response

Language version

No response

Manifest location and content before the Dependabot update

poetry.lock --- probably easiest to see the PR diff (linked below)

dependabot.yml content

See https://github.com/matrix-org/synapse/blob/develop/.github/dependabot.yml

Updated dependency

types-psycopg2, from 2.9.21.1 to 2.9.21.2

What you expected to see, versus what you actually saw

Expected:

• don't alter the dependency metadata in poetry.lock forpackages other than types-psycopg2
• update the locked version of types-psycopg2

Actual:

• All references to importlib-metadata, importlib-resources, pkgutil-resolve-name, tomli, typed-ast, typing-extensions removed from [package.dependencies] sections
• importlib-resources, pkgutil_resolve_name and typed-ast no longer included in the locked environment
• types-psycopg2 successfully updated

Native package manager behavior

2022-11-28 18:14:56 ✔  $ poetry update types-psycopg2
Updating dependencies
Resolving dependencies... (1.0s)

Writing lock file

No dependencies to install or update

$ git diff
diff --git a/poetry.lock b/poetry.lock
index d9e4803a5..c09151ff4 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -1448,7 +1448,7 @@ python-versions = "*"
 
 [[package]]
 name = "types-psycopg2"
-version = "2.9.21.1"
+version = "2.9.21.2"
 description = "Typing stubs for psycopg2"
 category = "dev"
 optional = false
@@ -1790,6 +1790,7 @@ constantly = [
 cryptography = [
     {file = "cryptography-38.0.3-cp36-abi3-macosx_10_10_universal2.whl", hash = "sha256:984fe150f350a3c91e84de405fe49e688aa6092b3525f407a18b9646f6612320"},
     {file = "cryptography-38.0.3-cp36-abi3-macosx_10_10_x86_64.whl", hash = "sha256:ed7b00096790213e09eb11c97cc6e2b757f15f3d2f85833cd2d3ec3fe37c1722"},
+    {file = "cryptography-38.0.3-cp36-abi3-macosx_10_12_universal2.whl", hash = "sha256:726e3a1bfee0e919b278c8f766fdcf1fe30f8e6feea590e3f248d3636b58ffb3"},
     {file = "cryptography-38.0.3-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:bbf203f1a814007ce24bd4d51362991d5cb90ba0c177a9c08825f2cc304d871f"},
     {file = "cryptography-38.0.3-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:554bec92ee7d1e9d10ded2f7e92a5d70c1f74ba9524947c0ba0c850c7b011828"},
     {file = "cryptography-38.0.3-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b1b52c9e5f8aa2b802d48bd693190341fae201ea51c7a167d69fc48b60e8a959"},
@@ -2813,8 +2814,8 @@ types-pillow = [
     {file = "types_Pillow-9.3.0.1-py3-none-any.whl", hash = "sha256:79837755fe9659f29efd1016e9903ac4a500e0c73260483f07296bd6ca47668b"},
 ]
 types-psycopg2 = [
-    {file = "types-psycopg2-2.9.21.1.tar.gz", hash = "sha256:f5532cf15afdc6b5ebb1e59b7d896617217321f488fd1fbd74e7efb94decfab6"},
-    {file = "types_psycopg2-2.9.21.1-py3-none-any.whl", hash = "sha256:858838f1972f39da2a6e28274201fed8619a40a235dd86e7f66f4548ec474395"},
+    {file = "types-psycopg2-2.9.21.2.tar.gz", hash = "sha256:bff045579642ce00b4a3c8f2e401b7f96dfaa34939f10be64b0dd3b53feca57d"},
+    {file = "types_psycopg2-2.9.21.2-py3-none-any.whl", hash = "sha256:084558d6bc4b2cfa249b06be0fdd9a14a69d307bae5bb5809a2f14cfbaa7a23f"},
 ]
 types-pyopenssl = [
     {file = "types-pyOpenSSL-22.1.0.2.tar.gz", hash = "sha256:7a350e29e55bc3ee4571f996b4b1c18c4e4098947db45f7485b016eaa35b44bc"},

Images of the diff or a link to the PR, issue, or logs

matrix-org/synapse#14558

Rich diff of the delta that dependabot proposed:

Smallest manifest that reproduces the issue

No response

[DMRobertson]

Apologies for the lack of minimal reproduction example.

We also saw this in matrix-org/synapse#14561, matrix-org/synapse#14562, matrix-org/synapse#14563, and matrix-org/synapse#14564.

In matrix-org/synapse#14558 (comment) I speculated that this might be related to the recent release of poetry-core---but that could just be a coincidence: I'm not sure what version of poetry and poetry-core dependabot uses behind the scenes. It kinda feels like an upstream bug though---why would dependabot be removing things from [package.dependencies] sections?

Edit: poetry pins the version of poetry-core that it depends on; #6226 (comment) has a much more plausible hypothesis.

[cvitan]

Seeing the same issue in multiple PR's/repos, common denominator seems to be that all [package.dependencies] with markers python_version or python_full_version are removed.

[dimbleby]

it's speculation because I don't know how dependabot works, but I'd be suspicious of #6191

what you are seeing seems to be consistent with dependabot-poetry solving as if for a python 3.11 project, regardless of what is actually in your pyproject.toml

[caugner]

Same issue here: mdn/yari@13a4352

[caugner]

@jeffwidman Would you have a chance to look into whether this might be a regression caused by #6191, as speculated above? 🙏

[jeffwidman]

👋 Sorry for the bug... clearly affecting a lot of folks. Myself or @pavera or someone else on the team will try to dig into this tomorrow, although as y'all know could take a few days to fix... won't know til we've looked into it.

[pavera]

what you are seeing seems to be consistent with dependabot-poetry solving as if for a python 3.11 project, regardless of what is actually in your pyproject.toml

@dimbleby the issue is almost certainly #6191 I'm working on a better fix there.

From the 2 repos linked above both specify a python = "^3.7" in pyproject.toml, dependabot attempts to use the "latest python that satisfies the python version specification" which in this case is 3.11. The change in #6191 now temporarily writes this new version back to pyproject.toml for cases where the project may specify a specific version and we need to temporarily ignore a patch version. IE a project specifies python = "3.10.6" but we want to run under 3.10.8, we are attempting to modify that requirement to python = "~3.10". We'll need to detect that the existing requirement is satisfied by 3.11 before overwriting I think as putting ~3.11 into the file changes poetry's resolution logic.

[pavera]

I've now deployed the fix for this issue. I'm going to close the issue for now, but if anyone sees that this is still occurring please re-open.

[DMRobertson]

We've just asked dependabot to recreate the PR I reported in the issue and the diff looks good to me. Just wanted to drop a quick note to confirm all looks well.

Many thanks for the quick response!