Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fixes for platform, pypy wheel builds #297

Merged
merged 7 commits into from
Feb 12, 2020
Merged

fixes for platform, pypy wheel builds #297

merged 7 commits into from
Feb 12, 2020

Conversation

mattip
Copy link
Collaborator

@mattip mattip commented Jan 21, 2020

Testing pypy-inside-docker revealed that the support for building a wheel using pypy inside manylinuxXXXX did not work. This PR:

  • adds a test
  • enables testing on other architectures under linux
  • refactors the pypy-specific install code, and moving it from osx_utils.sh to common_utils.sh
  • tries to work around some travis/docker problems discovered in numpy-wheels

@mattip
Copy link
Collaborator Author

mattip commented Jan 21, 2020

I tested this locally with

$cat buildscript.sh
set -x
export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
#export MB_PYTHON_VERSION=pypy3.6-7.3
export MB_ML_VER=2014 
export VENV=venv
export TEST_BUILDS=1
export ENV_VARS_PATH="tests/env_vars.sh"
source tests/test_multibuild.sh
$./buildscript.sh

and tried with a few different values for MB_PYTHON_VERSION

mattip
mattip commented Jan 21, 2020
View reviewed changes
.gitignore Outdated
xz-stamp
yaml*/
yaml-stamp
zlib-stamp
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.gitignore changes needed for local testing

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe *.stamp for neatness?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adopting

common_utils.sh
fi
PIP_CMD=pip
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved from osx_utils.sh and redone to allow pypy-5.4 or pypy3.6-7.3. Also properly sets PIP_CMD

common_utils.sh
$PYTHON_EXE -c "import sys; print('{0}.{1}'.format(*sys.version_info[0:2]))"
}


Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved from osx_utils.sh

configure_build.sh
@@ -10,6 +10,7 @@ fi
CONFIGURE_BUILD_SOURCED=1

BUILD_PREFIX="${BUILD_PREFIX:-/usr/local}"
MB_ML_VER=${MB_ML_VER:-1}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a default

configure_build.sh
fi
elif [ "${MB_PYTHON_VERSION:0:4}" == "pypy" ]; then
yum install -y wget
fi
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only install libtool on manylinux1. May need to revert this if testing shows it is needed. On manylinux2014, installing libtool also installs gcc. PyPy needs wget to download the tarball.

tests/test_multibuild.sh
@@ -40,6 +41,7 @@ if [ -n "$TEST_BUILDS" ]; then
source tests/test_manylinux_utils_docker.sh
source tests/test_library_builders.sh
"
build_multilinux $my_plat "pip install simplejson"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added test that exercises python and pip inside a docker environment

@mattip
Copy link
Collaborator Author

mattip commented Jan 22, 2020

All the macos builds are failing. I don't understand some of them, like the one where installing wheel fails because it is missing delocate?

/Library/Frameworks/Python.framework/Versions/3.5/bin/pip3.5 install -U \
                                                                                         wheel==0.32.0
Collecting wheel==0.32.0
  Downloading https://files.pythonhosted.org/packages/b3/bb/ \
      42354ce8c08f66ae0cd0f4a841f40ed41d709ac9c28f292bfeb383236a4a/ \
                                    wheel-0.32.0-py2.py3-none-any.whl
ERROR: delocate 0.8.0 requires bindepend, which is not installed.
ERROR: delocate 0.8.0 requires machomachomangler, which is not installed.
Installing collected packages: wheel
  Found existing installation: wheel 0.31.1
    Uninstalling wheel-0.31.1:
      Successfully uninstalled wheel-0.31.1

@matthew-brett
Copy link
Collaborator

Not sure what's going on - but maybe https://github.com/matthew-brett/multibuild/issues/298 ?

@mattip mattip force-pushed the fix-pypy branch 2 times, most recently from 5a0ae03 to aac4913 Compare January 22, 2020 22:42
@mattip mattip mentioned this pull request Jan 23, 2020
Closed
@mattip
Copy link
Collaborator Author

mattip commented Jan 27, 2020

Getting further. Now the macOS builds with TEST_BUILDS=1 are failing with Error: cfitsio 3.450_1 is already installed. Should this test be skipped on macOS?

The s390x failure is due to yum install failing, see nealef/clefos#1

@mattip
Copy link
Collaborator Author

mattip commented Jan 27, 2020

@matthew-brett: travis-ci build is green. Before I restore the appveyor CI file and ask for final review, could you take a look at this?

@mattip
Copy link
Collaborator Author

mattip commented Feb 5, 2020

@matthew-brett ping

matthew-brett
matthew-brett reviewed Feb 5, 2020
View reviewed changes
Copy link
Collaborator

@matthew-brett matthew-brett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup - seems reasonable.

.gitignore Outdated
xz-stamp
yaml*/
yaml-stamp
zlib-stamp
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe *.stamp for neatness?

@mattip
Copy link
Collaborator Author

mattip commented Feb 5, 2020

Watching the slow progress of the macOS builds on travis, it seems you are limited to two macOS simultaneous builds. There are 30 builds, each at 2 1/2 minutes so the macOS will take around 45 minutes of wall clock time. It might make sense to drop some of them or start to use azure

@mattip
Copy link
Collaborator Author

mattip commented Feb 5, 2020

@matthew-brett ready for final review. The s390x failure is a system thing: the yum mirror is flaky, see my discussion with them in nealef/clefos#1.

@mattip
Copy link
Collaborator Author

mattip commented Feb 12, 2020

@matthew-brett ping

@matthew-brett
Copy link
Collaborator

Sorry - yes - looks OK to me - let's give it a try.

@matthew-brett matthew-brett merged commit b943f33 into multi-build:devel Feb 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@matthew-brett matthew-brett matthew-brett left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mattip @matthew-brett