Make boost compatible with anaconda=4.2.0

[183amir]

I propose to add this into another branch called anaconda not master so that boost can be installed with anaconda=4.2.0.

right now installing boost in an environment with anaconda 4.2.0 installed gives this:

conda install boost=1.61.0
Using Anaconda API: https://api.anaconda.org
Fetching package metadata .........
Solving package specifications: ..........

Package plan for installation in environment /home/amir/miniconda/envs/test02:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    anaconda-navigator-1.2.3   |           py27_0         3.0 MB  defaults
    qtconsole-4.2.1            |           py27_0         132 KB  conda-forge
    spyder-2.3.9               |           py27_1         2.1 MB  conda-forge
    ------------------------------------------------------------
                                           Total:         5.3 MB

The following NEW packages will be INSTALLED:

    boost:              1.61.0-py27_1     conda-forge

The following packages will be UPDATED:

    anaconda:           4.2.0-np111py27_0 defaults    --> custom-py27_0     defaults   
    icu:                54.1-0            defaults    --> 56.1-4            conda-forge

The following packages will be SUPERCEDED by a higher-priority channel:

    pyqt:               5.6.0-py27_0      defaults    --> 4.11.4-py27_0     conda-forge
    qtconsole:          4.2.1-py27_1      defaults    --> 4.2.1-py27_0      conda-forge
    spyder:             3.0.0-py27_0      defaults    --> 2.3.9-py27_1      conda-forge

The following packages will be DOWNGRADED due to dependency conflicts:

    anaconda-navigator: 1.3.1-py27_0      defaults    --> 1.2.3-py27_0      defaults   
    matplotlib:         1.5.3-np111py27_0 defaults    --> 1.5.1-np111py27_0 defaults   
    qt:                 5.6.0-0           defaults    --> 4.8.7-3           defaults  

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[jakirkham]

Same story as with Jasper ( conda-forge/jasper-feedstock#11 ). Time for SONAMEs.

[183amir]

yes same story, but here I'm proposing a branch which will not break the pinning script. Can we get this merged in please until we have the sonames working?

[183amir]

@jakirkham I'm trying to release bob-2.4.0 in sync with anaconda-4.2.0 and this is the only thing that is blocking me. Could we have this as a separate branch please until SONAMEs and other solutions are in place?

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I was trying to look for recipes to lint for you, but it appears we have a merge conflict.
Please try to merge or rebase with the base branch to resolve this conflict.

Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

[jakirkham]

I was asleep, @183amir. Two pings along the same lines in the middle of the night seems excessive.

As far as this being the only blocker, don't you need JasPer too?

In any event, this has merge conflicts and was failing earlier. Maybe clean this up a bit first and we can talk.

Would be good if some other maintainers are on board also. So would like to wait a bit to give them time to respond.

Obviously can't merge this into master so we will need a new branch. Will worry about that after we sort out the other mentioned things.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[183amir]

As far as this being the only blocker, don't you need JasPer too?

No Jasper is not needed for bob. I need for my work though but it is not urgent and I can compile it locally.

I have rebased and tested locally. It works but let's wait for the CIs too.

[183amir]

OK tests are passing now (see second commit) please.

[183amir]

Any comment @conda-forge/boost ?

[jakirkham]

Probably shouldn't have skipped CI on the last commit, but I really doubt that it would have failed there if passed on the previous one.

[183amir]

Could you please merge this into a branch called anaconda? any other name would be fine also.

[183amir]

ping @jakirkham

[ccordoba12]

@183amir, I don't think it's a good thing to follow what you're proposing here. I mean, given that conda-forge and defaults diverged very early in their dependencies and that there's no necessity for both to have, for instance, the same versions of ICU, there's no reliable way to make packages in conda-forge to be installable with particular versions of Anaconda.

What users need to do is just to create conda envs and install packages from the conda-forge channel in them. Or if you want to provide those packages (as you're trying to do here), please make them available in your own channel. They don't need to be part of conda-forge :-)

[183amir]

@ccordoba12 I understand what you mean but you should know that conda-forge is a community channel and we are trying to provide something here for the community. The default way we advertise installing bob packages is just using the conda-forge channel: https://gitlab.idiap.ch/bob/bob/wikis/Installation

$ conda config --add channels conda-forge
$ conda create -n bob_env_py27 python=2.7 bob

But, sometimes things break in conda-forge and some packages are not as stable as they are in the defaults channel and we as researchers where we don't get paid to maintain packages on conda-forge and given the stress we always have in our work we just want something that is super stable, mkl enabled, and etc. to run our experiments with. This is why we decided to sync our bob releases with the anaconda package so that if needed we can create stable environments for our experiments. However, we always make sure that bob packages work with the conda-forge channel (we build and test them here).

please make them available in your own channel. They don't need to be part of conda-forge :-)

We had our own channel at the beginning but we saw something far greater here in conda-forge and decided to just use conda-forge and it has been an excellent choice for us. I have contributed back to the community too with more than 250 pulls so far. Now you are asking us besides maintaining 35+ packages here in conda-forge, we go and create our own channel again because I am asking for addition of one package with a different dependency which will not break anybody's workflow and just will allow more installation scenarios for users.

conda-forge is a community channel, we have to provide what community needs. I see that even some of core members are repinning packages and rebuilding them for themselves: conda-forge/jasper-feedstock#11 (comment) but WHY???!!! What is the point of working together and having a channel that is supposed to work for all of us?

I know we cannot satisfy everybody's needs but why stop people for making small changes that will satisfy their needs?

[183amir]

Also, the concept of having a package compiled several times with different dependencies is nothing new conda build does that automatically for numpy and I have been doing it for a couple of feedstocks for a while now: https://github.com/conda-forge/bob.io.image-feedstock/blob/master/conda-forge.yml, https://github.com/conda-forge/bob.io.base-feedstock/blob/master/conda-forge.yml, https://github.com/conda-forge/sox-feedstock/blob/master/conda-forge.yml

[ccordoba12]

@183amir, for your comments you seem a bit agitated, please calm down. I just dissented with you, I never said that

Now you are asking us besides maintaining 35+ packages here in conda-forge, we go and create our own channel again

I just said that you should use your own channel to provide Boost versions that are compatible with Anaconda, that's all. Or, you should ask Continuum (i.e. me because I maintain Boost in Continuum) to update Boost :-)

---

Look, there are two conflicting aims here: stable vs. bleeding edge. Anaconda and the packages it provides are more stable (as you said) but also a bit more outdated because Continuum is more conservative about updating them.

And conda-forge is all about bleeding edge: latest dependencies, always very up to date but with the drawback that things still break quite often.

So, in my humble opinion, you can't expect (or demand) both things to work together side by side because they pursue very different things.

---

Besides, I think you're not doing things correctly (if you want robustness and stability for your computations) by mixing packages compiled in conda-forge with packages compiled by Continuum (as you're doing in the packages you mentioned above). And here it is why: the compilers used in both systems are quite different on Linux and Mac.

1. On Linux:
   • Almost all Continuum packages are compiled in CentOS 5 with gcc 4.4 (and sometimes even with gcc 4.1).
   • Conda-forge packages are compiled in CentOS 6 with gcc 4.8 (if I'm not mistaken).
2. On Mac:
   • Continuum compiles almost everything with the gcc version that comes XCode (i.e. gcc 4.2) and so their packages are linked against libstdc++ (perhaps the only exception is Qt5).
   • Conda-forge uses clang and so their packages are compiled against libc++.

So, as you can see, either you live in Continuum's world or in conda-forge's one. There's no way to live in both at the same time and expect that things won't break in very obscure and bizarre ways.

---

So, to ratify what I said before, I'm -1 on this change. I hope I have been clearer this time :-)

[183amir]

@ccordoba12 sorry for being angry. I know what you mean but I also know what I am doing too.

On Linux, packages compiled in the defaults channel are compatible with packages in conda-forge. There is no problem with that. I use and test them all the time and we use Linux to run experiments anyway.

On Mac, what you are saying is true but that is still ok because we don't want to have super stable installation on mac's because eventually we run things on Linux for large experiments.

What I want from you and conda-forge community in general is to allow me to make small non-breaking changes in the channel without questioning what am I going to do with this compiled package even if I will do something wrong with it. I don't have another channel and nor want to have another one. My channel is conda-forge and changes that I am proposing here are valid and as I said nothing new conceptually (just like building with several numpy versions).

[ccordoba12]

Ok, these are my answers to your remarks:

On Linux, packages compiled in the defaults channel are compatible with packages in conda-forge.

Nop, that's not true, as I said above. The fact that you haven't encountered problems so far is just a matter of pure luck (or maybe because you have only dealt with C instead of C++ libraries).

If you want to be truly compatible with the packages provided by Continuum, you need to follow Continuum build specifications. They are clearly described here:

https://github.com/ContinuumIO/anaconda-recipes/blob/master/README.md

All Continuum employees who help to build packages follow those specifications, and you should do it too. Since those specs are very different from the conda-forge ones, you can't build packages here and said to your users that you're "compatible with Anaconda".

What you are really doing is preventing conda to complain because of conflicting dependencies between the default channels and the conda-forge one. That's a clever trick, but something that I think is fundamentally wrong.

What I want from you and conda-forge community in general is to allow me to make small non-breaking changes in the channel ...

What you're trying to do here is an important change and something that would affect all conda-forge users because after this change all its users (and not just yours) will be able to install the Boost conda-forge package along with the default ones.

... without questioning what am I going to do with this compiled package ...

Sorry, but that's why every package has a list of maintainers, instead of allowing all conda-forge members to touch all packages :-) And that is a good thing after all. I guess you wouldn't be happy if I'd come some day to the packages you maintain and try to modify them as I'd like :-)

... even if I will do something wrong with it.

So, if you're going to do something wrong, why not use your own channel instead of messing with conda-forge?

I don't have another channel and nor want to have another one.

Sorry, but conda-forge is not a private channel, it's a community channel and so we all have a saying about what you're trying to do :-)

I am proposing here are valid and as I said nothing new conceptually (just like building with several numpy versions).

I think you're wrong and don't understand what's the real issue here. Using different numpy versions has never implied mixing packages (or their dependencies) created under very different build specs.

If conda packages were versioned by build specs, then what you're trying to do is allowing users to install these packages:

 icu-54.1-centos5-gcc44
 boost-1.61-centos6-gcc48

and, as you can see, that doesn't make any sense. Your real choices are:

1. Conda-forge

    icu-56.1-centos6-gcc48
    boost-1.61-centos6-gcc48
   

2. Defaults

    icu-54.1-centos5-gcc44
    boost-1.61-centos5-gcc44
   

That's why I said yesterday that you can't live in both worlds at the same time (at least regarding to compiled packages :-)

---

Since I don't have more time to keep arguing about this, I finally want to say that I'm firmly against this change (because it doesn't make sense to me), and I propose this PR to be closed.

[183amir]

@ccordoba12 as a side request, could you please release boost-1.61.0 on the defaults channel?

Nop, that's not true, as I said above. The fact that you haven't encountered problems so far is just a matter of pure luck (or maybe because you have only dealt with C instead of C++ libraries).

They are compatible: conda-forge/conda-forge.github.io#115 (comment)

The only times I have seen things break is when you use a new gcc to compile and run with an older libgcc or mix the c++98/c++11 ABI. These are not the cases here. All packages are compiled with gcc 4.4 (defaults) or 4.8 (conda-forge) and are ran with libgcc 4.8.5 2 (a dependency of anaconda=4.2.0).

[ccordoba12]

could you please release boost-1.61.0 on the defaults channel?

I'll try to do it next week :-)

All packages are compiled with gcc 4.4 (defaults) or 4.8 (conda-forge)

True, but you're trying to mix them with this PR and (I think) they should be compiled and used with the same compiler and libgcc.

@mingwandroid, you have much more expertise with compilers than me. What do you think about what @183amir is trying to do here?

[mingwandroid]

This is an interesting PR. I'm against it, because if it were to be allowed (even if it does work, which it might up until someone tries to run on CentOS 5, or updates something somewhere) it opens the floodgates to attempting to workaround defaults and conda-forge incompatibilities in a very bad way. Already @183amir has said "but these guys do it over here" as justification (then the next person will point to this feedstock as justification for more of the same). Imagine the mess that would result if this became standard practice. What if someone wants to install bob and something that uses boost linked to icu-56 at the same time?

Can someone explain how to me how SONAMEs is intended to help out with this? Instead of libpng16 and libpng17 as in the proposal, are you contemplating something like boost-cf and boost-ac to denote conda-forge and anaconda-compatible versions respectively? IMHO SONAMEs logical conclusion is the manylinux so renaming procedure where the sha1 of the file itself is appended to it and patchelf is used to fix up the references.

From my observations, over time, these problems are growing, getting more complicated, splitting opinions and causing a lot of stress and the workarounds are horrible (which leads to more problems, more complication, more split opinions, more stress).

At root, it's all over a few minor package version differences with libpng, libjpeg, ncurses, libxml2 (or was it expat?) being the ones I see come up most frequently.

My opinion is that we need to get compatible, properly compatible, as in "these packages are (basically) the same thing so they cannot be incompatible" and then all this 'stuff' goes away. That's going to take a good CI/build-farm story and using the same set of compilers (and eventually, recipes). It will require compromises.

For now, @183amir if you want an Anaconda-compatible updated boost, then Continuum (Carlos and I) will be happy to sort that out for you. Its dependency tree and the environment in which is was built will contain nothing but Anaconda-compatible packages. I already have 1.61 built on all OSes, and it also now includes static libraries.

[ccordoba12]

it opens the floodgates to attempting to workaround defaults and conda-forge incompatibilities in a very bad way

Very good point! This could become bewildering in no time if feedstock after feedstock adopts this practice.

---

I'd also like to say that I'm very keen to this change because last year I had to fight for days a subtle bug in Boost serialization libraries, and the real cause was a compilation mismatch between Boost and Python (on Windows, no less!). Almost everything was working but serialization of C++ objects! So even if things seem to work, bugs like that one could be awaiting us around the corner ;-)

Finally, (although I shouldn't have to remark it) Boost is a critical piece for almost all modern C++ projects, being sort of its batteries-included standard library. So we can't risk to mess with it like this :-) Very complex projects (like Pygmo) are already built on top of it here in conda-forge :-)

[183amir]

@ccordoba12 and @mingwandroid thank you very much for detailed explanation but it now makes me more worried about stability than ever because sometimes because of dependency conflicts like boost you cannot mix packages from the defaults and conda-forge but usually there are no dependency conflicts and you can mix packages from defaults and conda-forge.

Now I am wondering how can we build packages within a community (not everyone doing it in their own channel) that would serve as a complementary set of packages to the defaults.

[ccordoba12]

usually there are no dependency conflicts and you can mix packages from defaults and conda-forge

Yep, there's no problem for pure Python packages but (as I said before) you can't basically mix them for compiled packages.

Now I am wondering how can we build packages within a community ... that would serve as a complementary set of packages to the defaults.

@183amir, this is the point I've been trying to make through this thread: conda-forge and defaults can't be complementary, they are different set of packages, and they should be treated that way. This means that they need to be installed in different environments and managed separately.

In a way, it's like the RedHat/CentOS relation: they are different distributions, each one with different rpm repositories.

[183amir]

@ccordoba12 would it be possible to add bob to the defaults channel? https://www.idiap.ch/software/bob/ https://github.com/conda-forge/bob-feedstock

[ccordoba12]

would it be possible to add bob to the defaults channel?

I'm not the one who decides that, sorry. But if we were to include it, we would be responsible of maintaining it, not you :-)

I think the best you can do is to create a VM with CentOS 5, use the recipes from conda-forge to create packages compatible with defaults and distribute bob using your own channel :-)

At least that's what I'd do ;-)

[ocefpaf]

IMO @ccordoba12 summarized it in:

So, in my humble opinion, you can't expect (or demand) both things to work together side by side because they pursue very different things.

And yet we do use them together in a complementary way in many places 😄
We do that by adding to conda-forge any package from defaults that needs to be re-build with conda-forge pinnings and relying on those that do not have such constraint.

The right solution is not to change conda-forge to conform, and like @mingwandroid said above badly, to defaults, nor to "force" defaults to update their packages. We should strive bring to conda-forge anything that needs re-pinning and ask continuum to update defaults when they see it is stable enough.

BTW in conda-forge/jasper-feedstock#12 (comment) @183amir said that all the package he needs are in conda-forge! So by asking for a pinning consistent to defaults is a "solve my particular case" kind of solution and not a community solution.

I do see the appeal in having some of conda-forge packages pinned like defaults, like in the Qt case, but only because conda-forge cannot yet build Qt! Still I never pushed for such solution because that would be disastrous. Instead of I pushing for an offline Qt build for conda-forge.

[183amir]

Thank you everyone for your comments. I know fully understand that conda-forge and defaults are not really as compatible as I want them to be. So the decisions to my problem which is stability (not dependency conflicts) boiled down to asking Continuum to add my packages to the defaults or maintaining these packages in my own channel.

Since adding them to the defaults channel would be difficult and may not happen and I don't want to solely maintain my own channel, I am now open to suggestions or volunteers to create another community led channel or even a label in conda-forge which would server to solely provide missing packages of the defaults channel and be 100% compatible with the defaults channel.

This way we can have bleeding edge packages in conda-forge which is very useful and have another channel/label to serve more stable (and probably older) builds.

[ocefpaf]

Thank you everyone for your comments. I know fully understand that conda-forge and defaults are not really as compatible as I want them to be.

Glad you understand.

Since adding them to the defaults channel would be difficult and may not happen and I don't want to solely maintain my own channel, I am now open to suggestions or volunteers to create another community led channel or even a label in conda-forge which would server to solely provide missing packages of the defaults channel and be 100% compatible with the defaults channel.

If you want consistency with defaults only for consistency sake I do not see how you can do that besides creating another channel. If you seek consistency with defaults because a package in conda-forge is broken then we should fix it. Otherwise why not do like the rest of us and use channel preference, and add defaults packages that are missing in conda-forge? With the exception of Qt this has worked out really nice so far.

[ccordoba12]

I agree with @ocefpaf: conda-forge should be completely self-contained and so we need to move to conda-forge all missing packages from defaults :-)

[183amir]

If you want consistency with defaults only for consistency sake I do not see how you can do that besides creating another channel.

I was thinking that maybe we can create another label in conda-forge and call it defaults? and the packages for this label are built in the feedstocks but in a different branch called defaults? and in that label we use the cent-os 5 docker image and in Mac link to libstdc++ (and do other necessary things to be compatible with defaults). This would require some changes to conda-smithy and a different toolchain and conda-forge-build-setup packages (which would live in the defaults label) but I think it is feasible. And of course, I would be willing to maintain this label and branches in the feedstocks. Also, we would have this only for the missing packages from the defaults channel.

[jakirkham]

TBH that sounds like a totally different community, @183amir. That isn't to say it doesn't have value and isn't worth pursing. It is just what you are asking for is a second set of pinnings, compilers used, docker images, and VMs (defaults uses Mac OS 10.7 for instance which Travis CI doesn't supply). This will also have unknowable effect on recipes (new patches, some packages can't be ported, etc.). This is just way out of scope for conda-forge and is a big ask for the community. Trying to keep such vast differences in conda-forge will have an increased maintenance cost for everyone involved and most likely be detrimental to both communities (defaults-compat and conda-forge). When this project was starting to really gain traction, there were a lot of analogies to the RedHat ecosystem. Along those lines, I would argue that defaults is RedHat and conda-forge is Fedora. What you are asking for sounds like CentOS (channel name TBD). At this stage, there is an undeniable cost (financial and otherwise) to making that happen (whether or not at conda-forge). Down the road that may be more achievable, but probably still at some cost as older OSes will need to be kept around for such ranges of compatibility. This all being said, I do understand why you ask and think it is desirable. Also I can see that some of the maintenance services for such a community could borrow heavily from what has been done here at conda-forge. Down the road maybe it could borrow from whatever build infrastructure is come up with at conda-forge. There will probably be close involvement between both communities on all these points. It just doesn't seem like an easy ask for conda-forge to bifurcate internally in this way. If someone sees it another way, please feel free to correct me.

cc @pelson

[ccordoba12]

Things will be better once we (i.e. Continuum) drop support for CentOS 5, which reaches end of life next year :-)