Skip to content

discord-fixes-20210722

commit 42433aa61ae4e54d3e24d736228c16b65ccf4a0d
Author: Danny Weinberg <FuegoFro@gmail.com>
Date:   Fri Jul 16 00:21:58 2021 +0000

    Use `_permissive_temp_dir` in more places.

commit b28c667627ab17fb6d7f5801dd542cf25c20039f
Author: Danny Weinberg <FuegoFro@gmail.com>
Date:   Fri Jul 16 00:13:36 2021 +0000

    Ignore `PermissionError`s explicitly, since `ignore_errors` only ignores `OSError`s

commit ae2cfc81fb8d9c08d17fcae33ebc4c76d88b790c
Author: Danny Weinberg <FuegoFro@gmail.com>
Date:   Fri Jul 16 00:05:44 2021 +0000

    Ignore errors deleting temp repo, make test param name better

commit 57138304132abe6f596e60b5fac26057509add1d
Author: Danny Weinberg <FuegoFro@gmail.com>
Date:   Thu Jun 10 22:46:05 2021 +0000

    Faster no-ops for VCS dependencies

    This allows pip-sync to no-op faster for VCS dependencies that haven't changed. Before this change when running `pip-sync` with a requirements file that has VCS packages it would always uninstall and reinstall those dependencies. This is because `pip` primarily operates on versions, and assumes that different code will have a different version. However this assumption breaks down with VCS packages, in which two different commits may very well have different code but the same version. Therefore to be safe we historically have had to reinstall these VCS dependencies.

    This extends `pip-sync` to write some extra metadata to the installed package to be able to track what revision the package was installed from, and then skip the uninstall-and-reinstall if the revision that we want to install is the same as the revision that is already installed. This significantly speeds up no-op `pip-sync` invocations for requirements that have multiple VCS packages.

    The tests added are written as best I could for unit tests. I think a better test would likely be to spin up an entirely new virtualenv, do a sync in it, and verify that a subsequent sync doesn't reinstall anything, but the exsting tests seem less like full integration tests so I didn't go down that route. As is, there's a somewhat tricky bit of logic that reaches into the workings of `pip` that's mocked out in the tests (namely the `_reload_installed_distributions_by_key` function).

    Given a `requirements.in` of

    ```
    -e git+https://github.com/python/mypy.git@v0.902#egg=mypy
    -e git+https://github.com/psf/black.git@21.5b2#egg=black
    -e git+https://github.com/yaml/pyyaml.git@5.4.1.1#egg=pyyaml
    ```

    and a corresponding `requirements.txt` of

    ```
    -e git+https://github.com/psf/black.git@21.5b2#egg=black
    -e git+https://github.com/python/mypy.git@v0.902#egg=mypy
    -e git+https://github.com/yaml/pyyaml.git@5.4.1.1#egg=pyyaml
    appdirs==1.4.4
    click==8.0.1
    importlib-metadata==4.5.0
    mypy-extensions==0.4.3
    pathspec==0.8.1
    regex==2021.4.4
    toml==0.10.2
    typed-ast==1.4.3
    typing-extensions==3.10.0.0
    zipp==3.4.1
    ```

    I ran tests to determine how this affects the fresh install time as well as the no-op install time. I also did the tests but with non-editable VCS dependencies, both with the tag and with an exact commit hash (since pip has optimizations for the latter case):

    | Environment                                          | Fresh install time (approx seconds) | No-op install time (approx seconds) |
    | ---------------------------------------------------- | ----------------------------------- | ----------------------------------- |
    | Editable, without optimization                       | 20                                  | 12                                  |
    | Editable, with optimization                          | 20 🟡                               | 0.5 🟢                              |
    | Non-editable, without optimization                   | 20                                  | 20                                  |
    | Non-editable, with optimization                      | **25** 🔴                           | 5 🟢                                |
    | Non-editable with exact commit, without optimization | 3                                   | 2                                   |
    | Non-editable with exact commit, with optimization    | **7** 🔴                            | **4** 🔴                            |

    As you can see, most of the number remain similar (for fresh installs) or significantly better (for no-op installs). There are some regressions in a few places, namely due to the extra time it takes to calculate the relevant information, both when detecting no-ops before installation and when writing that into the environment after installation.
Assets 2