Add manylinux2014 draft

pep-9999.rst

@@ -0,0 +1,317 @@
+PEP: 9999
+Title: The manylinux2014 Platform Tag
+Version: $Revision$
+Last-Modified: $Date$
+Author: Dustin Ingram <dii@google.com>
+BDFL-Delegate: Nick Coghlan <ncoghlan@gmail.com>
+Discussions-To: Distutils SIG <distutils-sig@python.org>
+Status: Active
+Type: Informational
+Content-Type: text/x-rst
+Created: 29-April-2019
+Post-History: 29-April-2019
+
+
+Abstract
+========
+
+This PEP proposes the creation of a ``manylinux2014`` platform tag to
+succeed the ``manylinux2010`` tag introduced by :pep:`513`. It also
+proposes that PyPI and ``pip`` both be updated to support uploading,
+downloading, and installing ``manylinux2014`` distributions on
+compatible platforms.
+
+Rationale
+=========
+
+CentOS 6 is now the oldest supported CentOS release, and will receive
+maintenance updates through November 30th, 2020, [1]_ at which point
+it will reach end-of-life, and no further updates such as security
+patches will be made available. All wheels built under the
+``manylinux2010`` images will remain at obsolete versions after that
+point.
+
+Therefore, we propose the continuation of the existing manylinux
+standard, and that a new PEP 425-style [2]_ platform tag called
+``manylinux2014`` be derived from CentOS 7 and that the ``manylinux``
+toolchain, PyPI, and ``pip`` be updated to support it.
+
+Similar to how :pep:`571` and :pep:`513` drew allowed shared
+libraries and their symbol versions from CentOS 5.11 and CentOS 6,
+respectively, a ``manylinux2014`` platform tag will draw its libraries
+and symbol versions from CentOS 7, which will reach end-of-life on
+June 30th, 2024. [1]_
+
+The ``manylinuxYYYY`` pattern has a number of advantages that motivate
+continuing with the current status quo:
+
+- Well-defined Docker images with clearly specified compatible
+  libraries;
+- No need to survey for compatibility issues across multiple releases;
+- A single build image and ``auditwheel`` profile per architecture.
+
+There are also some disadvantages:
+
+- Requires drafting a new PEP for every new standard;
+- Requires adding the new platform tag to installers (e.g., ``pip``);
+- Installers are unable to install a platform tag which predates a
+  given release.
+
+There are also challenges which would exist for any proposal,
+including the time and effort it takes to define, prepare and release
+the Docker images and corresponding ``auditwheel`` profiles. These
+challenges were experienced in the long rollout period for
+``manylinux2010``, which took approximately 1 year from PEP acceptance
+to compatible build environment published. [3]_
+
+However, if this PEP can be an indicator, the process is now
+well-defined and easily repeatable, which should increase the timeline
+for rollout of a newer, updated platform tag.
+
+The ``manylinux2014`` policy
+============================
+
+The following criteria determine a ``linux`` wheel's eligibility for
+the ``manylinux2014`` tag:
+
+1. The wheel may only contain binary executables and shared objects
+   compiled for one of the following architectures supported by CentOS
+   7, or a CentOS 7 compatible base image (such as ubi7): [4]_ ::
+
+       x86_64
+       i686
+       aarch64
+       armhfp
+       ppc64
+       ppc64le
+       s390x
+
+   This list adds support for ARM (aarch64, armhfp) and PowerPC
+   (ppc64, ppc64le) architectures supported by the CentOS Alternative
+   Architecture Special Interest Group, as well as the IBM Z (s390x)
+   architecture. [5]_
+
+2. The wheel's binary executables or shared objects may not link
+   against externally-provided libraries except those in the following
+   list: ::
+
+       libgcc_s.so.1
+       libstdc++.so.6
+       libm.so.6
+       libdl.so.2
+       librt.so.1
+       libcrypt.so.1
+       libc.so.6
+       libnsl.so.1
+       libutil.so.1
+       libpthread.so.0
+       libresolv.so.2
+       libX11.so.6
+       libXext.so.6
+       libXrender.so.1
+       libICE.so.6
+       libSM.so.6
+       libGL.so.1
+       libgobject-2.0.so.0
+       libgthread-2.0.so.0
+       libglib-2.0.so.0
+
+   This list is identical to the externally-provided libraries
+   allowed for ``manylinux2010``. ``libpythonX.Y`` remains ineligible
+   for inclusion for the same reasons outlined in :pep:`513`.
+
+   On Debian-based systems, these libraries are provided by the
+   packages:
+
+   ============  =======================================================
+   Package       Libraries
+   ============  =======================================================
+   libc6         libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6,
+                 libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1,
+                 libnsl.so.1
+   libgcc1       libgcc_s.so.1
+   libgl1        libGL.so.1
+   libglib2.0-0  libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0
+   libice6       libICE.so.6
+   libsm6        libSM.so.6
+   libstdc++6    libstdc++.so.6
+   libx11-6      libX11.so.6
+   libxext6      libXext.so.6
+   libxrender1   libXrender.so.1
+   ============  =======================================================
+
+   On RPM-based systems, they are provided by these packages:
+
+   ============  =======================================================
+   Package       Libraries
+   ============  =======================================================
+   glib2         libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0
+   glibc         libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1,
+                 libcrypt.so.1, libpthread.so.0, libdl.so.2, libm.so.6,
+                 libc.so.6
+   libICE        libICE.so.6
+   libX11        libX11.so.6
+   libXext:      libXext.so.6
+   libXrender    libXrender.so.1
+   libgcc:       libgcc_s.so.1
+   libstdc++     libstdc++.so.6
+   mesa          libGL.so.1
+   ============  =======================================================
+
+3. If the wheel contains binary executables or shared objects linked
+   against any allowed libraries that also export versioned symbols,
+   they may only depend on the following maximum versions::
+
+       GLIBC_2.17
+       CXXABI_1.3.7
+       GLIBCXX_3.4.19
+       GCC_4.8.5
+
+   As an example, ``manylinux2014`` wheels may include binary
+   artifacts that require ``glibc`` symbols at version ``GLIBC_2.12``,
+   because this an earlier version than the maximum of ``GLIBC_2.17``.
+4. If a wheel is built for any version of CPython 2 or CPython
+   versions 3.0 up to and including 3.2, it *must* include a CPython
+   ABI tag indicating its Unicode ABI.  A ``manylinux2014`` wheel
+   built against Python 2, then, must include either the ``cpy27mu``
+   tag indicating it was built against an interpreter with the UCS-4
+   ABI or the ``cpy27m`` tag indicating an interpreter with the UCS-2
+   ABI. [6]_ [7]_
+5. A wheel *must not* require the ``PyFPE_jbuf`` symbol.  This is
+   achieved by building it against a Python compiled *without* the
+   ``--with-fpectl`` ``configure`` flag.
+
+Compilation of Compliant Wheels
+===============================
+
+Like ``manylinux1``, the ``auditwheel`` tool adds ``manylinux2014``
+platform tags to ``linux`` wheels built by ``pip wheel`` or
+``bdist_wheel`` in a ``manylinux2014`` Docker container.
+
+Docker Images
+-------------
+
+A ``manylinux2014`` Docker image based on CentOS 7 x86_64 is provided
+for building binary ``linux`` wheels that can reliably be converted to
+``manylinux2014`` wheels. [8]_ This image comes with a full compiler
+suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.5) as well as
+the latest releases of Python and ``pip``.
+
+Compatibility with kernels that lack ``vsyscall``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Similar to CentOS 6, CentOS 7 includes a version of ``glibc`` that
+depends on the ``vsyscall`` page. As described in :pep:`571`, this
+breaks the assumption that a Docker containers userland is compatible
+wtih its host's kernel. Similar to ``manylinux2010``, Docker images
+for ``manylinux2014`` require patching ``glibc`` to remove all
+dependencies on ``vsyscall`` in the version of ``glibc`` included with
+our Docker image.
+
+Auditwheel
+----------
+
+The ``auditwheel`` tool has also been updated to produce
+``manylinux2014`` wheels. [9]_ Its behavior and purpose are otherwise
+unchanged from :pep:`513`.
+
+Platform Detection for Installers
+=================================
+
+Platforms may define a ``manylinux2014_compatible`` boolean attribute
+on the ``_manylinux`` module described in :pep:`513`. A platform is
+considered incompatible with ``manylinux2014`` if the attribute is
+``False``.
+
+If the ``_manylinux`` module is not found, or it does not have the
+attribute ``manylinux2014_compatible``, tools may fall back to
+checking for glibc. If the platform has glibc 2.17 or newer, it is
+assumed to be compatible unless the ``_manylinux`` module says
+otherwise.
+
+Specifically, the algorithm we propose is::
+
+    def is_manylinux2014_compatible():
+        # Only Linux, and only supported architectures
+        from distutils.util import get_platform
+
+        if get_platform() not in [
+            "linux-x86_64",
+            "linux-i686",
+            "linux-aarch64",
+            "linux-armhfp",
+            "linux-ppc64",
+            "linux-ppc64le",
+            "linux-s390x",
+        ]:
+            return False
+
+        # Check for presence of _manylinux module
+        try:
+            import _manylinux
+
+            return bool(_manylinux.manylinux2014_compatible)
+        except (ImportError, AttributeError):
+            # Fall through to heuristic check below
+            pass
+
+        # Check glibc version. CentOS 7 uses glibc 2.17.
+        # PEP 513 contains an implementation of this function.
+        return have_compatible_glibc(2, 17)
+
+Backwards compatibility with ``manylinux2010`` wheels
+=====================================================
+
+As explained in :pep:`513`, the specified symbol versions for
+``manylinux1`` allowed libraries constitute an *upper bound*.  The
+same is true for the symbol versions defined for ``manylinux2014`` in
+this PEP.  As a result, ``manylinux1`` and ``manylinux2010`` wheels
+are considered ``manylinux2014`` wheels.  A ``pip`` that recognizes
+the ``manylinux2014`` platform tag will thus install ``manylinux2010``
+wheels for ``manylinux2014`` platforms -- even when explicitly set --
+when no ``manylinux2014`` wheels are available.
+
+PyPI Support
+============
+
+PyPI should permit wheels containing the ``manylinux2014`` platform
+tag to be uploaded in the same way that it permits ``manylinux2010``.
+It should not attempt to verify the compatibility of ``manylinux2014``
+wheels.
+
+References
+==========
+
+.. [1] CentOS Product Specifications
+   (https://wiki.centos.org/About/Product)
+.. [2] PEP 425 -- Compatibility Tags for Built Distributions
+   (https://www.python.org/dev/peps/pep-0425/)
+.. [3] Tracking issue for manylinux2010 rollout
+   (https://github.com/pypa/manylinux/issues/179)
+.. [4] Red Hat Universal Base Image 7
+   (https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/ubi7)
+   (https://wiki.centos.org/SpecialInterestGroup/AltArch)
+.. [5] The CentOS Alternative Architecture Special Interest Group
+   (https://wiki.centos.org/SpecialInterestGroup/AltArch)
+.. [6] PEP 3149
+   https://www.python.org/dev/peps/pep-3149/
+.. [7] SOABI support for Python 2.X and PyPy
+   https://github.com/pypa/pip/pull/3075
+.. [8] manylinux2014 Docker images
+   (https://quay.io/repository/pypa/manylinux2014_x86_64)
+.. [9] auditwheel
+   (https://github.com/pypa/auditwheel/)
+
+Copyright
+=========
+
+This document has been placed into the public domain.
+
+..
+   Local Variables:
+   mode: indented-text
+   indent-tabs-mode: nil
+   sentence-end-double-space: t
+   fill-column: 70
+   coding: utf-8
+   End: