How to make manylinux package with MKL dependency

[TonyXiang8787]

I am working on a Python package with C++ extensions which have Intel MKL dependency. The extension module links to libmkl_rt.so. I have put mkl>=2019 in the requirements in setup.py.

The MKL itself is a manylinux1 package in the PyPI:
https://pypi.org/project/mkl/#files

However, if I run auditwheel repair to produce a wheel file with manylinux2014 tag. It will copy the libmkl_rt.so into the wheel file. This is of course not needed (and also not preferred). The requirement on MKL is already specified and can be found in the PyPI library. And the auditwheel repair only copies the libmkl_rt.so which is merely an entry for the real MKL library.

Is it a intended behavior of auditwheel repair to copy all the external dynamic dependencies into the wheel file, even if the dependency is a manylinux1 package in the PyPI and the requirements are specified in setup.py? If yes, how I can avoid that the libmkl_rt.so when producing a manylinux2014 wheel?

[fzimmermann89]

Does anybody have an advice how to handle this?

[mayeut]

Is it a intended behavior of auditwheel repair to copy all the external dynamic dependencies into the wheel file

Yes, it's the intended behavior of auditwheel. It grafts all external dynamic dependencies into the wheel file (with the exception of those listed in the whitelist) for it to be installable on any manylinux compliant system.

even if the dependency is a manylinux1 package in the PyPI and the requirements are specified in setup.py?

That's an undefined behavior. PyPI is only there to manage python dependencies, not native ones and thus, auditwheel does not rely on such information. The fact that Intel uses PyPI to redistribute libraries is either for convenience or for some legal reason (e.g. the license forbids redistribution or modification).

auditwheel will modify libraries when they're grafted in

If yes, how I can avoid that the libmkl_rt.so when producing a manylinux2014 wheel?

You can remove the library from the needed libraries and have a mechanism to dynamically load libmkl_rt.so before you import your extension. That's the way onnxruntime does it for CUDA.

[TonyXiang8787]

@mayeut thanks for your answer. We will try to dynamically load mkl at runtime.

This issue can be closed I think.