Python 3.9 Support

[johnthagen]

Tracking issue for Python 3.9 support, set to release Oct 5, 2020.

Python 3.9rc1 has been released so it is possible to test in a near-final state. Travis CI python3.9-dev exists that could be used to test builds.

[skvark]

Different Python versions are provided by the manylinux images and multibuild and 3.9 is probably available also in them. Travis build host Python is not used in the builds or tests.

[johnthagen]

Python 3.9 released today.

[skvark]

Yeah, manylinux2014 Docker images in this repo need to be rebuilt when the base images have been updated from rc2 to the final version: https://github.com/pypa/manylinux/blob/master/docker/build_scripts/build_env.sh

[skvark]

I'll do the needed changes and rebuilds today, manylinux and multibuild were just updated to the 3.9 official release version.

Also we need to wait for numpy 1.19.3 release which should available in a week or so.

[skvark]

Waiting also for this: appveyor/ci#3541

Branch with the changes: https://github.com/skvark/opencv-python/tree/feat/python39

[skvark]

numpy/numpy#17482

[cclauss]

https://github.com/numpy/numpy/releases/tag/v1.19.3 Delivers numpy Python 3.9 binary wheels on all supported platforms.

[skvark]

Travis builds look ok, but I have to think what to do with Appveyor. Python 3.9 was installed only on the Visual Studio 2019 images (appveyor/ci#3541 (comment)) and that's not going to work for this project. Currently, Visual Studio 2015 is used for building the wheels. scikit-build does not recognize VS 2019 as a valid build tool so I'll have to install Python 3.9 manually to the Appveyor environment before builds. Upgrading to VS 2019 would also affect to the VC++ redistributable version needed to run these packages.

[skvark]

Release builds are now in progress but it will take over 24 hours before all wheels have been built and uploaded to PyPI.

[skvark]

Some of the wheels are not yet uploaded, opencv-python-headless hit the PyPI project size limit and prevents some builds from uploading new wheels. I'll delete some of the oldest releases from PyPI today to make room for new ones.

[skvark]

I don't understand why some projects in PyPI have 20 GB limit, some 30 GB limit and some 40 GB limit... maybe something to do with the time of creation ¯_(ツ)_/¯ I uploaded the missing opencv-python-headless wheels manually from my Azure artifact storage to PyPI. Windows builds are still running and will hopefully finish today.

[johnthagen]

pypa/packaging-problems#86 (comment)

We can increase the limit on a package per package basis.

It looks like they will happily increase the limit for legitimate packages if you asked. To me, opencv-python seems like a very good candidate due to how many precompiled wheels are needed to host it properly.

Looks like you can ask here if you wanted: https://github.com/pypa/warehouse/issues

[skvark]

Yeah, I have already individual package size limit requests in place (source distribution for contrib variants are missing due to this): #412

However, I will also probably need to request limit increases for the projects in PyPI. Current status for different projects:

opencv-python

Project size : 28.4 GiB
Project total size limit : 40.0 GiB

opencv-contrib-python

Project size : 30.6 GiB
Project total size limit : 40.0 GiB

opencv-python-headless:

Project size : 19.5 GiB
Project total size limit : 20.0 GiB

opencv-contrib-python-headless:

Project size : 24.0 GiB
Project total size limit : 30.0 GiB

[cclauss]

Staggering! They are massive.

[skvark]

In addition to the PyPI artifacts, I'm storing currently about 150 GB of macOS and Linux build artifacts in Azure. Appveyor luckily provides native free artifact storage.

I think the wheels have a reasonable size (around 20-50 MB per each depending on platform), there are just a lot of releases and maybe 20 wheels in each of them.

Have a look for example at tensorflow, single wheel size is several hundred MB: https://pypi.org/project/tensorflow/#files

[skvark]

pypi/support#712

[mshabunin]

There is PYTHON3_LIMITED_API option which enables building a single binary for all python 3.x versions (>=3.4) (opencv/opencv#14736). But numpy must be >=1.17 or <1.15. And I'm not sure how it aligns with all toolchains, manylinux etc.

[skvark]

That limited API sounds great if it works reliably. Numpy can be forced to >=1.17 but this needs also other investigation and testing. Added issue so that I don't forget it: #414