Satisfy both conda-forge and defaults channel requirements

[183amir]

No description provided.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

• Failed to even lint the recipe (might be a conda-smithy bug) 😢

[ocefpaf]

@183amir I'd rather rebuilt anything that is linking to the old jpeg with the new one rather than this. Otherwise we might get in trouble trying to support multiple jpegs everywhere.

I believe that the main issue right now is Qt, right? @msarahan how far are we from having a new qt with the latest jpeg?

[183amir]

The problem is that I want to install anaconda=4 and that comes with jpeg-8d. We have diverged a lot from the defaults channel.

[183amir]

Why did we switch to jpeg9b anyway?

[msarahan]

how far are we from having a new qt with the latest jpeg?

I think this is more constrained by politics than by technical concerns. I think we could have working qt4 and qt5 recipes very soon - certainly by the end of the week. However, I do not feel that the question of hosting recipes that are not (currently) built with CI has been settled.

[ocefpaf]

However, I do not feel that the question of hosting recipes that are not (currently) built with CI has been settled.

Got it. I am 👍 to a semi-obscure 😈 build for qt as long as the jpeg issue is resolved.

[jakirkham]

Why could Qt packages not be added to defaults?

[msarahan]

They are in defaults - but not with JPEG 9 support. You either need to build Qt here, or switch to JPEG 9 there. I don't think I have reasonable justification for pushing the latter.

[ocefpaf]

They are in defaults - but not with JPEG 9 support. You either need to build Qt here, or switch to JPEG 9 there. I don't think I have reasonable justification for pushing the latter.

Agreed. Conda-forge should be providing a qt with jpeg 9 and let defaults follow its own timetable.

[jakirkham]

Personally, I'm still not sold on hosting packages that we are not building with some sort of build bot. I know that there are other community members that feel very strongly about this. So, don't really want to see this proceed without discussing it in a meeting.

[jakirkham]

Alright, it seems that moving forward on different jasper builds is the least controversial of the options. Are we ok to see @183amir continue this course of action?

[ocefpaf]

My opinion is on #9 (comment). Please do not merge this until more members say that are willing to support build matrix for jpeg everywhere or want to work on fixing qt.

[jakirkham]

This is not my favorite solution either, but the other two potential resolutions are feeling very controversial right now. However, this feels less controversial. I expect @183amir is only wanting to do this to a few packages (JasPer, OpenCV 2.x, anything else?). If we can live with this sort of thing in a couple of places, it at least let's him go forward with getting something installable in Anaconda. That way we can give the other more controversial suggestions more time to simmer.

[183amir]

@ocefpaf I don't want to use the jpeg-9 stack because it does not align with the defaults channel. This has nothing to do with qt. As long as the defaults channel is using jpeg-8 I want to use it too.
And as for members willing to support build matrices, this is a community effort and people who use jpeg-8 (me) will make sure that the packages they need has a jpeg-8 variant. There is no need to build the whole stack with jpeg-8.

[183amir]

Also, the conda-forge channel is not as stable as I would like it to be right now. So, I am trying to pull minimal packages from conda-forge as possible when creating an environment and supporting jpeg-8 would help me do that. Example: https://travis-ci.org/conda-forge/bob.io.image-feedstock/builds/135645811 https://github.com/bioidiap/bob.io.image/issues/13 conda-forge/xz-feedstock#7

[ocefpaf]

@ocefpaf I don't want to use the jpeg-9 stack because it does not align with the defaults channel.

That kind of beats the purpose of conda-forge. Sorry to heat that.

This has nothing to do with qt. As long as the defaults channel is using jpeg-8 I want to use it too.

I understand. I had the same problem with a many updated software added in conda-forge that did not align with defaults, but I worked to get everything updated rather than taking a step back. If we had just added jpeg 9 I would understand, but everything in conda-forge is using jpeg 9 now.

What else from defaults to you need with jpeg 8 that is not qt? Can you add those to conda-forge?

@msarahan do you think it is possible to build qt with jpeg 9 using continuum resources/machines and upload to conda-forge? Since we already use qt built that way from defaults I cannot see how that is controversial.

Also, the conda-forge channel is not as stable as I would like it to be right now.

I know. I am working on the matplotlib+gdal problem more than a month now and I do want to move forwards and not backwards. We as a community should avoid the "works for me" kind of solution. Downgrading jpeg would solve some of my problems too, but I don't think it is what conda-forge should do as the better solution of updating the rest of the of the stack to jpeg 9 is not so far way.

[ocefpaf]

Also, I asked for patience here. There are any solutions of the kind I need this that will bring headache in the long term. The PR here, for example, needs a linter update and some discussion on opening a precedent to adding another matrix build. That should be discussed in the meeting too!

Note that I am waiting for more than a month for the qt+jpeg solution, can we wait a little bit more please?

ping: @gillins, @pelson, @msarahan, and @kmuehlbauer here.

[msarahan]

@msarahan do you think it is possible to build qt with jpeg 9 using continuum resources/machines and upload to conda-forge?

Sure. Between @ccordoba12 and I, we have already built many platforms. Need to check JPEG version on them, though. Would be nice to finalize the staged-recipe PRs to have a solid reference point. We can rebuild these pretty quickly - maybe 1-2 hours apiece?

[ccordoba12]

What other things an update to jpeg 9 will break for us?

[ccordoba12]

Us -> Continuum :-)

[msarahan]

Carlos - I don't think anything breaks - just all the packages have to be lined up. So we need a complete set of solutions here, such that nothing in defaults that uses JPEG 8 is installed.

Longer-term, we need to add versions to package names and SO names, so that we can have multiple versions installed. We talked about this in this past Friday's meeting.

[ocefpaf]

Sure. Between @ccordoba12 and I, we have already built many platforms. Need to check JPEG version on them, though.

That would be awesome! Even better if you could the same Qt recipe in conda-forge/staged-recipes#706 and conda-forge binaries for the dependencies.

@ccordoba12 this hypothetical qt would be uploaded to conda-forge only to avoid messing up defaults. (Just in case that was not clear above.)

[ocefpaf]

Aside: I am 👍 on having SO names, but did anyone really though about where that will lead us? condaOS 😈, condaDist 😺, or condaApp 🙀

[ccordoba12]

We'll try to update to jpeg 9 this week. A Qt4 package based on conda-forge/staged-recipes#706 and built with conda-forge deps will have to wait two or three more weeks, I'm afraid.

We're really busy right now trying to move things to Qt5 for Anaconda 4.1 :-)

[ocefpaf]

We'll try to update to jpeg 9 this week. A Qt4 package based on conda-forge/staged-recipes#706 and built with conda-forge deps will have to wait two or three more weeks, I'm afraid.

Thanks! I thought that the former solution would be harder, but that works too.

We're really busy right now trying to move things to Qt5 for Anaconda 4.1 :-)

Looking forward to it!

[ocefpaf]

Why did we switch to jpeg9b anyway?

Missed that comment. There are many reasons, but this is the main one.

[183amir]

If anacodna-4.1 is going to be released with jpeg-9, I'm ok with using jpeg-9.

[jakirkham]

Missed that comment. There are many reasons, but this is the main one.

Thanks for clarifying. I think I was out when most of this happened.

[ccordoba12]

If anacodna-4.1 is going to be released with jpeg-9, I'm ok with using jpeg-9.

Sorry guys, after a bit of chatting we decided to leave the move to Jpeg 9 to be done after Anaconda 4.1. We still have a lot to do for that release :-)

[jakirkham]

Understood, thanks for letting us know, @ccordoba12. 😄

[183amir]

Sorry guys, after a bit of chatting we decided to leave the move to Jpeg 9 to be done after Anaconda 4.1. We still have a lot to do for that release :-)

If that is the case, can we go forward with this pull then? we can revert the matrix once the defaults channel is moved to jpeg-9 as well. I want jpeg-8 builds for only jasper and opencv.

[183amir]

Downgrading jpeg would solve some of my problems too, but I don't think it is what conda-forge should do as the better solution of updating the rest of the of the stack to jpeg 9 is not so far way.

This PR does not defeat that purpose. It will still keep jpeg-9 going while enabling jpeg-8 too.

Let me explain my situation a bit more. I have created an environment with anaconda 4.0 and have ran some experiments with it for a paper. Now, I want to run some extra experiments which require opencv but opencv is built with jpeg-9 which is much higher than what the defaults channel provides. Now, I don't want to update my whole stack (which will potentially change the results of my old experiments) just to install opencv.