Add `name` key to `sources` in `Pipfile`s #7744

jeffwidman · 2023-08-06T06:32:34Z

Newer versions of pipenv require any specified sources to be
explicitly name'd.

More context here: pypa/pipenv#5370 (comment)

This has been true since at least Sept 2022. Our version of pipenv is
from April of 2022, so it doesn't complain. But it will as soon as we upgrade.

Also, for sources that dynamically injects into the
Pipfile, they need a name. These sources are stripped from the final
Pipfile / Pipfile.lock during the FileUpdater#post_process_lockfile
method, so all we need is a placeholder name to placate pipenv.

Long term, we may want to add custom error handling to flag this
missing key as a Dependabot::DependencyFileNotResolvable error.

But I decided that was out of scope for now as this PR does not generate
the error... that will not happen until we upgrade to newer pipenv.
And even at that point, our first priority will be upgrading, and then
from there handling any new errors that start popping up.

And even then, most of the active users of pipenv are unlikely to see this
error because they're likely running a newer version of pipenv.

jeffwidman · 2023-08-06T06:34:09Z

Likely there will be hash mismatches in the fixture lockfiles, so they'll need regenerating... if so, probably best to only do what's necessary to get past CI... as no point in regenerating in this PR using an older version of pipenv only to have to regenerate them again under the newer version of pipenv... and probably best to wait until we switch to pipenv upgrade so that we don't have to regenerate yet again:

jeffwidman · 2023-08-10T03:06:45Z

python/spec/fixtures/pipfile_files/version_not_specified.lock

@@ -1,12 +1,13 @@
 {
    "_meta": {
        "hash": {
-            "sha256": "c402ea48092e9d467af51a483bb8dd8ad0620e11c94f009dcd433f97a99d45db"
+            "sha256": "e76ae491d793d659f05c7f7eab261fd6167dc062efcba08f17be68e73eb87665"


This was the only lockfile that the tests required to have an updated hash.

I strongly suspect the hashes are outdated for the other fixture lockfiles, but again, we can regenerate them all after updating to latest pipenv once we drop Python 3.7.

python/lib/dependabot/python/file_updater/pipfile_preparer.rb

Newer versions of `pipenv` require any specified `sources` to be explicitly `name`'d. More context here: pypa/pipenv#5370 (comment) This has been true since at least Sept 2022. Our version of `pipenv` is from April of 2022, so it doesn't complain. But it will as soon as we upgrade. Also, for sources that

dynamically injects into the `Pipfile`, they need a name. These sources are stripped from the final `Pipfile` / `Pipfile.lock` during the `FileUpdater#post_process_lockfile` method, so all we need is a placeholder `name` to placate `pipenv`. Long term, we may want to add custom error handling to flag this missing key as a `Dependabot::DependencyFileNotResolvable` error. But I decided that was out of scope for now as this PR does not generate the error... that will not happen until we upgrade to newer `pipenv`. And even at that point, our first priority will be upgrading, and then from there handling any new errors that start popping up. And even then, most of the active users of `pipenv` are unlikely to see this error because they're likely running a newer version of `pipenv`.

jeffwidman · 2023-08-29T20:53:13Z

We got a report via a support ticket that opened a PR that included index: dependabot-inserted-index-0 in Pipfile.lock. So this PR introduced some bug...

Unfortunately, I can't repro and the affected user's repo is a private company so I can't access their repo and they are migrating to a different python package manager so decided it wasn't worthwhile for them to try to create a reproducible example.

The one observation they made:

the bad index seems to be added while updating the sub-dependencies of our private libs.

If anyone else encounters this, please feel free to file a new issue, preferably here in dependabot-core and linking to reproducible example and I'd be happy to take a look.

juanitosvq · 2023-08-30T16:20:56Z

We are having the same issue but unfortunately also in a private repository 😞

It looks like older versions of pipenv are happy with that new index, but certainly the newest version pipenv-2023.8.28 is not:

Successfully uninstalled pipenv-2023.2.4

..............................................................................................................................

Installing dependencies from Pipfile.lock (923e0b)...
--
125 | Unable to find dependabot-inserted-index-0 in sources, please check
126 | dependencies:

We also have private libraries, so not sure I'll be able to create a reproducible example.

jeffwidman · 2023-08-31T00:51:04Z

@juanitosvq Can you at least create a new issue describing what you're seeing? The pipenv maintainers are great folks and we have good lines of communication open with them, so maybe between all of us we can get it figured out...

But this isn't a conversation we should be having on an already merged PR... 😁

Newer versions of `pipenv` require any specified `sources` to be explicitly `name`'d. More context here: pypa/pipenv#5370 (comment) This has been true since at least Sept 2022. Our version of `pipenv` is from April of 2022, so it doesn't complain. But it will as soon as we upgrade. Also, for sources that

dynamically injects into the `Pipfile`, they need a name. These sources are stripped from the final `Pipfile` / `Pipfile.lock` during the `FileUpdater#post_process_lockfile` method, so all we need is a placeholder `name` to placate `pipenv`. Long term, we may want to add custom error handling to flag this missing key as a `Dependabot::DependencyFileNotResolvable` error. But I decided that was out of scope for now as this PR does not generate the error... that will not happen until we upgrade to newer `pipenv`. And even at that point, our first priority will be upgrading, and then from there handling any new errors that start popping up. And even then, most of the active users of `pipenv` are unlikely to see this error because they're likely running a newer version of `pipenv`.

github-actions bot added the L: python label Aug 6, 2023

jeffwidman added the Ecosystems Used by the maintainer team for internal-facing project tracking label Aug 6, 2023

jeffwidman self-assigned this Aug 6, 2023

jeffwidman force-pushed the add-missing-name-key-to-sources branch 5 times, most recently from 9b638cf to 62f4d33 Compare August 10, 2023 00:48

jeffwidman marked this pull request as ready for review August 10, 2023 00:49

jeffwidman requested a review from a team as a code owner August 10, 2023 00:49

jeffwidman force-pushed the add-missing-name-key-to-sources branch 5 times, most recently from 883d742 to d97b36b Compare August 10, 2023 03:05

jeffwidman commented Aug 10, 2023

View reviewed changes

jeffwidman force-pushed the add-missing-name-key-to-sources branch from d97b36b to 26d416e Compare August 10, 2023 17:24

jeffwidman commented Aug 10, 2023

View reviewed changes

python/lib/dependabot/python/file_updater/pipfile_preparer.rb Outdated Show resolved Hide resolved

deivid-rodriguez approved these changes Aug 10, 2023

View reviewed changes

python/lib/dependabot/python/file_updater/pipfile_preparer.rb Show resolved Hide resolved

jeffwidman force-pushed the add-missing-name-key-to-sources branch 2 times, most recently from ae4e7bd to dd9672c Compare August 10, 2023 19:01

jeffwidman changed the title ~~Add name key to sources~~ Add name key to Pipfile's sources Aug 10, 2023

jeffwidman changed the title ~~Add name key to Pipfile's sources~~ Add name key to sources in Pipfiles Aug 10, 2023

jeffwidman force-pushed the add-missing-name-key-to-sources branch 2 times, most recently from 7b0c471 to 781896c Compare August 10, 2023 19:06

jeffwidman force-pushed the add-missing-name-key-to-sources branch from 781896c to 16be5d4 Compare August 10, 2023 19:12

jeffwidman enabled auto-merge (squash) August 10, 2023 19:13

jeffwidman disabled auto-merge August 10, 2023 19:14

jeffwidman merged commit 918333f into main Aug 10, 2023
90 checks passed

jeffwidman deleted the add-missing-name-key-to-sources branch August 10, 2023 19:40

juanitosvq mentioned this pull request Aug 31, 2023

Dependabot adding index to Pipfile.lock causes issue in newest version of pipenv #7936

Open

1 task

juanitosvq mentioned this pull request Nov 30, 2023

Dependabot adding index to Pipfile.lock causes issue in newest version of pipenv #8496

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `name` key to `sources` in `Pipfile`s #7744

Add `name` key to `sources` in `Pipfile`s #7744

jeffwidman commented Aug 6, 2023 •

edited

Loading

jeffwidman commented Aug 6, 2023 •

edited

Loading

jeffwidman Aug 10, 2023

jeffwidman commented Aug 29, 2023

juanitosvq commented Aug 30, 2023

jeffwidman commented Aug 31, 2023 •

edited

Loading

Add name key to sources in Pipfiles #7744

Add name key to sources in Pipfiles #7744

Conversation

jeffwidman commented Aug 6, 2023 • edited Loading

jeffwidman commented Aug 6, 2023 • edited Loading

jeffwidman Aug 10, 2023

Choose a reason for hiding this comment

jeffwidman commented Aug 29, 2023

juanitosvq commented Aug 30, 2023

jeffwidman commented Aug 31, 2023 • edited Loading

Add `name` key to `sources` in `Pipfile`s #7744

Add `name` key to `sources` in `Pipfile`s #7744

jeffwidman commented Aug 6, 2023 •

edited

Loading

jeffwidman commented Aug 6, 2023 •

edited

Loading

jeffwidman commented Aug 31, 2023 •

edited

Loading