Proposal: split the package into dev and binary

[jankatins]

[This came out of the last conda-forge hangout + original proposal; this proposal is only for native packages, not python/... ones]

Problem

The problem is that packages end up being incompatible between versions ("SONAME change"). This is currently not the "real" problem with libpng (but with jpeg -> 8 vs 9). To guard against this (future potential) problem, the dependency to this package is pinned to libpng 1.6*, so that packages linking against this packages can't install a future libpng 1.7. The problem with that is that in some cases (aka the jpeg package) different packages in the archive (in this cases matplotlib in conda-forge and pyqt in defaults) depend on different versions of the library and can't therefore be installed together.

The solution is to make multiple versions of a package installable next to each other. This is possible because the actual binary library libpng16.{so|dll} has the version in the name and therefore two versions can be installed next to each other. Unfortunately, the current package also contains the header files (png.h and also the *lib files on windows) and these are unversioned. Therefore, to not overwrite files from another packages, the header files need to be separated from the binary file.

proposal

(roughly modeled after the way debian does it):

• add two new packages/feedstocks libpng16-dev and libpng16 which are named after the complete SONAME of the *.so/*.dll file in the package (name+version). The recipes are basically a copy of the current recipe with one packages deleting everything but the one binary *so/*dll file (libpng16) and the other deleting only the binary (libpng16-dev).
• libpng-dev needs to runtime depend on the same version as the libpng16 package to pull in the right so/dll at compile time -> at compile time the same file content as the current libpng package is present on the file system -> this needs coordinated uploads for libpng16 and libpng16-dev
• changing all dependencies to build-depend on libpng16-dev >=1.6.21 (the current minimum compatible version) and the runtime dependency to ``libpng16 >=1.6.21. This only needs greater than` dependencies as well behaving libraries should change the SONAME on incompatible ABI/API changes and so the package would need to depend on `libpng17{,-dev}`

The result of these is that the current libpng and libpng16 will be coinstallable but overwrite the actual binary lib (_so/_dll), but this can't be worked around without changing the SONAME/Filename on disk (not a good idea...) or having the packages conflict with each other (which conda currently can't do?!). The idea is to do the split and then do a rebuild of all the conda-forge packages to get the new version until no package in conda-forge depends on libpng anymore.

Bad things which can still happen:

• if all libpng dependent packages in conda-forge get rebuild and if libpng gets removed from conda-forge, then a package which gets installed from defaults alone will pull in libpng again but this package will overwrite the libpng16.so/dll file and so an incompatible version would be present. For that I would suggest that an empty libpng (with a specific higher version, which the libpng package in defaults would not reach) would be kept in the archive and libpng16 would depend on that version until defaults gets updated to this new split libpng package. On the other hand, this can lead to the situation that a package from the defaults channel would depend on libpng without a * pin or a less than restriction (haven't investigated how the default packages declare their dependency on libpng) and would end up without a libpng16.so/dll :-(
• potentially two different versions of the dev packages could be installed. Conda currently does not complain about overwriting files in a different package. Unfortunately, this would make linking against a *so/*dll nondeterministic as the last package installed would "win" the link. This needs conflicts in the libpng16-dev package (and stuff like virtual packages -> libpng16-dev would providelibpng-devand conflict withlibpng-dev-> only one package providinglibpng-dev` can be installed). But unless someone screws up a recipe and the CI services/conda-build install both versions I don't think it will matter.
• other problems?

"Future directions" :-)

• The same proposal should be used for jpeg and the other native packages. jpeg is actually easier because AFAIK only conda-forge has the incompatible jpeg9 version and defaults is on jpeg8 -> no compatibility is needed, we just need to remove the old conda-forge jpeg package
• if conda gets multiple binary packages per recipe, then libpng16-dev and libpng16 should be build from the same source and the version can be set in one recipe. Until then it needs coordinated uploads.
• if conda gets dynamic build time runtime dependency (= the actual dependencies of the runtime package are generated at build time from the actual versions of the "linked against" packages), then the build time dependency could be relaxed to libpng-dev/libpng16-dev (-> without a specific version; without the version in the name only if the API is kept and only the ABI is changed -> the recipe would be call "libpng" for all libpng versions) -> whatever version is present would end as the runtime version dependency and all is fine :-)
• Conflicts would make the "one package could overwrite the *so/*dll with an older version"-problem much easier: We could then declare a conflict in libpng16 against libpng and force that package out.
• if conda would error when a file from another package is overwritten would make the problem of libpng from defaults overwriting the so/dll from libpng16 from conda-forge at least detectable on install time and not on runtime.

[jankatins]

Ah, first problem: the jpeg includes both an unversioned libjpeg.dll and some *.exe files.

• The dll without the version should be renamed via a patch or we need a conflicts feature in conda? so files shouldn't be a problem because the file is named libjpeg.so.8
• The *.exe would either need to go to the dev file or to a separate third libjpeg-progs (debians name) package (I've no idea what the binaries are for).

Update

On windows it looks like jpeg 8 runtime dependencies are not needed as jpeg 8 only includes static .lib files (not dlls + implib), so the jpeg stuff is statically compiled into the package.

C:\portabel\miniconda\pkgs\pyqt-4.11.4-py35_5\Lib\site-packages\PyQt4
λ dumpbin.exe /DEPENDENTS *pyd |grep jpeg

-> It seems that packages from defaults, at least on windows, do not need to depend on jpeg.

So this won't overwrite and we have one less problem.

For the change of libjpeg.dll -> libjpeg9.dll: s/libpng/libpng9/g in https://github.com/conda-forge/jpeg-feedstock/blob/master/recipe/CMakeLists.txt.patch and add a compat copy for the implib file?

CC: @gillins @jakirkham (jpeg maintainer)

[jankatins]

Any opinions here?

[msarahan]

I wish I could line comment.

the other deleting only the binary (libpng16-dev).
libpng-dev needs to runtime depend

Are these the same? In my mind, there should only be libpng-dev, and its version should capture the SO version. It should, as you say, imply a particular libpngSO..

There are a lot of good ideas here. I think you should raise many of them on the conda and conda-build issue trackers.

[jakirkham]

I'm sorry @JanSchulz I have not had the time to give this the attention it deserves. Maybe it will be more effective to discuss it again. Perhaps in a topic meeting dedicated to this?

[jankatins]

In my mind, there should only be libpng-dev, and its version should capture the SO version. It should, as you say, imply a particular libpngSO

Mostly the idea is to either use libsthX-dev and only greater than dependencies OR libsth-dev and *. For build time dependencies * dependencies are probably fine, the only pain would be when jpeg would update the version in a way that all pinnings would need to be updated. Not sure if that would be possible if the pinning is on 9*...

I'm currently thinking about submitting two recipes for jpeg9 and jpeg-dev to demonstrate the concept. Especially since windows jpeg 8 hasn't even a dll file included and so wouldn't have a potential problem with overwritten dll files :-)

[jakirkham]

Maybe this would be a good CFEP. It's basically there AFAICT. Some formatting would need to be done, but it would give this idea greater visibility.

[ocefpaf]

@jankatins now that conda supports outputs we can revisit this.
However I'd rather we open a new issue and coordinate with defaults to ensure consistency between the recipes.