Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pip-sync does not verify hashes when installing packages #619

Closed
jdufresne opened this issue Dec 6, 2017 · 2 comments
Closed

pip-sync does not verify hashes when installing packages #619

jdufresne opened this issue Dec 6, 2017 · 2 comments
Labels
PR wanted Feature is discussed or bug is confirmed, PR needed

Comments

@jdufresne
Copy link
Member

Describe the issue briefly here.

If a hash is incorrect, pip will fail with a command like:

THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    django==2.0.0 from https://pypi.python.org/packages/44/98/35b935a98a17e9a188efc2d53fc51ae0c8bf498a77bc224f9321ae5d111c/Django-2.0-py3-none-any.whl#md5=da2fdc3901e8751aa7835f49fb6246b2 (from -r requirements.txt (line 7)):
        Expected sha256 6914851d4a7ff8cbd32b73c6076441f377c45a5bbff7e771798fb02c43c31f47
        Expected     or af18618ce3291be5092893d8522fe3916961bf3a1fb60e3858ae74865a4f07c2
             Got        af18618ce3291be5092893d8522fe3919661bf3a1fb60e3858ae74865a4f07c2

However, pip-sync will not fail. It will happily install the package even if the hashes do no match. I expect pip-sync to also fail if it can't verify the package hashes.

Environment Versions
  1. Linux -- Fedora 27
  2. Python version: 3.6.3
  3. pip version: 9.0.1
  4. pip-tools version: 1.11.0
Steps to replicate
  1. Change the hashes in requirements.txt so they are obviously wrong
  2. Run pip-sync requirements.txt in a fresh virtualenv
Expected result

pip-sync fails with a loud warning that the hashes do not match (like pip)

Actual result

pip-sync installs the packages with mismatched hashes.

I have written a test script to demonstrate. In this script, pip-sync installs packages with mismatched hashes. At the end, the test is rerun with pip to demonstrate what I believe should happen.

requirements.in:

Django==2.0.0

test.sh:

#!/bin/bash

set -e
set -x

# Compile the requirements.in
rm -f requirements.txt
python3 -m venv venv
. venv/bin/activate
pip install pip-tools
venv/bin/pip-compile --generate-hashes -o requirements.txt requirements.in
deactivate

# Change the Django hash
sed -i -e 's/96/69/g' requirements.txt

# Demonstrate that pip-sync does not fail
rm -rf venv
python3 -m venv venv
. venv/bin/activate
pip install pip-tools
pip-sync requirements.txt
deactivate

# Confirm pip fails
rm -rf venv
python3 -m venv venv
. venv/bin/activate
pip install pip-tools
pip install -r requirements.txt
deactivate

Full script output:

$ ./test.sh 
+ rm -f requirements.txt
+ python3 -m venv venv
+ . venv/bin/activate
++ deactivate nondestructive
++ '[' -n '' ']'
++ '[' -n '' ']'
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/home/jon/test/venv
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/home/jon/.nvm/versions/node/v7.3.0/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/jon/.composer/vendor/bin:/home/jon/.local/bin:/home/jon/bin:/home/jon/devel/scripts
++ PATH=/home/jon/test/venv/bin:/home/jon/.nvm/versions/node/v7.3.0/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/jon/.composer/vendor/bin:/home/jon/.local/bin:/home/jon/bin:/home/jon/devel/scripts
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ '[' 'x(venv) ' '!=' x ']'
++ PS1='(venv) '
++ export PS1
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
+ pip install pip-tools
Requirement already satisfied: pip-tools in ./venv/lib/python3.6/site-packages
Requirement already satisfied: click>=6 in ./venv/lib/python3.6/site-packages (from pip-tools)
Requirement already satisfied: first in ./venv/lib/python3.6/site-packages (from pip-tools)
Requirement already satisfied: six in ./venv/lib/python3.6/site-packages (from pip-tools)
Requirement already satisfied: setuptools in ./venv/lib/python3.6/site-packages (from pip-tools)
+ venv/bin/pip-compile --generate-hashes -o requirements.txt requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes --output-file requirements.txt requirements.in
#
django==2.0.0 \
    --hash=sha256:9614851d4a7ff8cbd32b73c6076441f377c45a5bbff7e771798fb02c43c31f47 \
    --hash=sha256:af18618ce3291be5092893d8522fe3919661bf3a1fb60e3858ae74865a4f07c2
pytz==2017.3 \
    --hash=sha256:c41c62827ce9cafacd6f2f7018e4f83a6f1986e87bfd000b8cfbd4ab5da95f1a \
    --hash=sha256:fae4cffc040921b8a2d60c6cf0b5d662c1190fe54d718271db4eb17d44a185b7 \
    # via django
+ deactivate
+ '[' -n /home/jon/.nvm/versions/node/v7.3.0/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/jon/.composer/vendor/bin:/home/jon/.local/bin:/home/jon/bin:/home/jon/devel/scripts ']'
+ PATH=/home/jon/.nvm/versions/node/v7.3.0/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/jon/.composer/vendor/bin:/home/jon/.local/bin:/home/jon/bin:/home/jon/devel/scripts
+ export PATH
+ unset _OLD_VIRTUAL_PATH
+ '[' -n '' ']'
+ '[' -n /bin/bash -o -n '' ']'
+ hash -r
+ '[' -n '' ']'
+ unset VIRTUAL_ENV
+ '[' '!' '' = nondestructive ']'
+ unset -f deactivate
+ sed -i -e s/96/69/g requirements.txt
+ rm -rf venv
+ python3 -m venv venv
+ . venv/bin/activate
++ deactivate nondestructive
++ '[' -n '' ']'
++ '[' -n '' ']'
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/home/jon/test/venv
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/home/jon/.nvm/versions/node/v7.3.0/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/jon/.composer/vendor/bin:/home/jon/.local/bin:/home/jon/bin:/home/jon/devel/scripts
++ PATH=/home/jon/test/venv/bin:/home/jon/.nvm/versions/node/v7.3.0/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/jon/.composer/vendor/bin:/home/jon/.local/bin:/home/jon/bin:/home/jon/devel/scripts
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1='(venv) '
++ '[' 'x(venv) ' '!=' x ']'
++ PS1='(venv) (venv) '
++ export PS1
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
+ pip install pip-tools
Collecting pip-tools
  Using cached pip_tools-1.11.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./venv/lib/python3.6/site-packages (from pip-tools)
Collecting click>=6 (from pip-tools)
  Using cached click-6.7-py2.py3-none-any.whl
Collecting first (from pip-tools)
  Using cached first-2.0.1-py2.py3-none-any.whl
Collecting six (from pip-tools)
  Using cached six-1.11.0-py2.py3-none-any.whl
Installing collected packages: click, first, six, pip-tools
Successfully installed click-6.7 first-2.0.1 pip-tools-1.11.0 six-1.11.0
+ pip-sync requirements.txt
Collecting django==2.0.0
  Using cached Django-2.0-py3-none-any.whl
Collecting pytz==2017.3
  Using cached pytz-2017.3-py2.py3-none-any.whl
Installing collected packages: pytz, django
Successfully installed django-2.0 pytz-2017.3
+ deactivate
+ '[' -n /home/jon/.nvm/versions/node/v7.3.0/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/jon/.composer/vendor/bin:/home/jon/.local/bin:/home/jon/bin:/home/jon/devel/scripts ']'
+ PATH=/home/jon/.nvm/versions/node/v7.3.0/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/jon/.composer/vendor/bin:/home/jon/.local/bin:/home/jon/bin:/home/jon/devel/scripts
+ export PATH
+ unset _OLD_VIRTUAL_PATH
+ '[' -n '' ']'
+ '[' -n /bin/bash -o -n '' ']'
+ hash -r
+ '[' -n '(venv) ' ']'
+ PS1='(venv) '
+ export PS1
+ unset _OLD_VIRTUAL_PS1
+ unset VIRTUAL_ENV
+ '[' '!' '' = nondestructive ']'
+ unset -f deactivate
+ rm -rf venv
+ python3 -m venv venv
+ . venv/bin/activate
++ deactivate nondestructive
++ '[' -n '' ']'
++ '[' -n '' ']'
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/home/jon/test/venv
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/home/jon/.nvm/versions/node/v7.3.0/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/jon/.composer/vendor/bin:/home/jon/.local/bin:/home/jon/bin:/home/jon/devel/scripts
++ PATH=/home/jon/test/venv/bin:/home/jon/.nvm/versions/node/v7.3.0/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/jon/.composer/vendor/bin:/home/jon/.local/bin:/home/jon/bin:/home/jon/devel/scripts
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1='(venv) '
++ '[' 'x(venv) ' '!=' x ']'
++ PS1='(venv) (venv) '
++ export PS1
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
+ pip install pip-tools
Collecting pip-tools
  Using cached pip_tools-1.11.0-py2.py3-none-any.whl
Collecting first (from pip-tools)
  Using cached first-2.0.1-py2.py3-none-any.whl
Collecting six (from pip-tools)
  Using cached six-1.11.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./venv/lib/python3.6/site-packages (from pip-tools)
Collecting click>=6 (from pip-tools)
  Using cached click-6.7-py2.py3-none-any.whl
Installing collected packages: first, six, click, pip-tools
Successfully installed click-6.7 first-2.0.1 pip-tools-1.11.0 six-1.11.0
+ pip install -r requirements.txt
Collecting django==2.0.0 (from -r requirements.txt (line 7))
  Using cached Django-2.0-py3-none-any.whl
Collecting pytz==2017.3 (from -r requirements.txt (line 10))
  Using cached pytz-2017.3-py2.py3-none-any.whl
THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    django==2.0.0 from https://pypi.python.org/packages/44/98/35b935a98a17e9a188efc2d53fc51ae0c8bf498a77bc224f9321ae5d111c/Django-2.0-py3-none-any.whl#md5=da2fdc3901e8751aa7835f49fb6246b2 (from -r requirements.txt (line 7)):
        Expected sha256 6914851d4a7ff8cbd32b73c6076441f377c45a5bbff7e771798fb02c43c31f47
        Expected     or af18618ce3291be5092893d8522fe3916961bf3a1fb60e3858ae74865a4f07c2
             Got        af18618ce3291be5092893d8522fe3919661bf3a1fb60e3858ae74865a4f07c2
@vphilippon
Copy link
Member

@jdufresne Thanks for the report, and good catch!

I took a quick look at the code: pip-sync does a pip install pkg_a pkg_b ..., not a pip install -r requirements.txt, which is the cause of the issue.

I feel like this could be changed, which could simplify pip-sync and align it more with pip. That would deserve a look.

@jdufresne
Copy link
Member Author

That makes sense to me. So, IIUC, pip-sync's job would be to:

  1. Remove all extraneous packages installed to site-packages
  2. Rerun pip install -r requirements.txt to install all missing packages

Yeah, I like that.

@vphilippon vphilippon added the PR wanted Feature is discussed or bug is confirmed, PR needed label Dec 19, 2017
@suutari suutari mentioned this issue Jan 2, 2018
Open
@atugushev atugushev mentioned this issue Jan 6, 2019
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
PR wanted Feature is discussed or bug is confirmed, PR needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jdufresne @vphilippon