Downgrading to JPEG 8

[jakirkham]

Having JPEG 9 has cause no end of grief lately. Would it be possible for us to downgrade to JPEG 8 so as to match defaults? This would allow our stuff and qt to live in the same environment.

cc @pelson @ocefpaf @kmuehlbauer @183amir

[ocefpaf]

Having JPEG 9 has cause no end of grief lately.

I worked a lot on the packages I am a member of to support jpeg and, as far as I know, qt is the last missing piece. Why not put our efforts on having qt with jpeg 9?

[jakirkham]

Unfortunately Qt is not going to build on the CIs. It takes way too long. Why do we need JPEG 9? I know we upgraded, but I don't think I ever understood clearly why it was necessary.

[ocefpaf]

Unfortunately Qt is not going to build on the CIs

Let's discuss more that and settle on a solution. I see a lot of value in moving forward with the qt recipe even if we don't build on our CIs.

[jankatins]

An alternative would be to add a jpeg9 package (or Dev + binary) and rebuild on that.

[183amir]

@ocefpaf but why do you need jpeg-9?

[jakirkham]

Honestly, I think part of the pain in doing that upgrade was it happened manually and took quite a bit of work to track down everything effected. However, we have far better tools for this problem now.

I think it is totally reasonable for people to be able to install matplotlib with stuff from here, but right now that is simply not possible because of this discontinuity around jpeg versions.

[183amir]

I would like to mention we have the same problem with libpng too. Defaults comes with 1.6.17 and we build with 1.6.21 and they are incompatible.

[jakirkham]

It's true. I can't remember if there were any other libraries like that, but I do remember libpng was a problem too.

[jakirkham]

Alright, this is clearly quite controversial and not going to happen in the near term. Let's see what acceptable workarounds we can do for the near term.

[jakirkham]

Not trying to make this any more controversial, just trying to note our decisions are affecting downstream users. In some cases, they don't know their downstream yet. For instance, see this break at docker-stacks that affects an R user. ( jupyter/docker-stacks#210 ) Given how willing Jupyter has been to adopt conda-forge, we should keep in mind how to avoid causing breaks for them IMHO. I'm guessing we still don't have R ready here ( though I haven't looked at that recently conda-forge/staged-recipes#586 ). When do have the full stack that defaults has, this will be less of a problem, but until then this is something to be aware of.

[patricksnape]

@jakirkham Which CIs are taking too long for Qt? Can we request more time for those CI builds?

[jakirkham]

Sorry, this discussion has been happening at two places, which makes it a bit hard to follow. See this PR ( conda-forge/jasper-feedstock#9 ) for the other half. It sounds like Continuum will switch to JPEG 9, which will make this less of an issue.

[jakirkham]

@patricksnape

Here's a PR ( conda-forge/staged-recipes#706 ) for Qt4 that is just building on CircleCI. It is getting right up next to the CircleCI limit, but it fits. Definitely doesn't fit on Travis CI. I don't recall what the status was with AppVeyor.

From what @msarahan has said Qt5 is probably not going to work on CI. I don't know how long the build takes; so, you would need to ask him. Here is a PR ( conda-forge/staged-recipes#706 ). However, it is missing some dependencies.

We certainly can always ask for more build time. We would need a better idea of what to ask for. The long term plan is to move to our own build infrastructure. I don't know if this will need some mixture of CIs too or what at this point. This is really still in the idea phase. This would require things like joining NumFOCUS, which thus far people here seem agreeable to. Also, it seems NumFOCUS is interested in working with us. So that will certainly open a path for this transition, which we ultimately want.

[ocefpaf]

@patricksnape at PyCon @jjhelmus, @scopatz, and I had a nice chat with some numfocus people and there is a path to create a conda-forge infrastructure. After PyCon that I had a nice chat with @kwilcox, who will probably attend our next meeting, and he has some very nice ideas on how to achieve that goal.

However, asking for more CI time right now is definitely the way to go! Any other solution will take at best 1-2 months to materialize and conda-forge moves faster than that.

Do you want to take charge of writing Travis-CI and ask for more CI time? (Not sure if we will get anything from AppVeyor as we are already on their best option.)

[jakirkham]

Do we know how long it takes to build Qt5, @msarahan and @ccordoba12? Would be nice to ask for a certain (reasonable) amount of build time. 😄

[ccordoba12]

I can only answer for Linux and Mac: Qt5 takes between 2 to 3 hours on a old (2012) Core i5 using 4 cores.

But the problem is not only time. CircleCI imposes a 4gb limit on space, and that's way too short for Qt5 (it needs 8 to 10gb).

[jakirkham]

Travis CI also has a memory limit of 3GB so that is something we need to ask about too. Though to try to make it easier on them, we can try to ask this for one or two feedstocks instead of the whole flock (pl?).

[ccordoba12]

@jakirkham, then let's merge the Qt4 PR and use it to ask for more time/space/memory :-)

The Qt5 one is not ready yet.

[jakirkham]

How long/much does it need?

[183amir]

while we are at it, can we ask for more time in opencv too, please?

[jakirkham]

Why do we need more time for OpenCV?

[ccordoba12]

Qt4 is easier: It needs like one hour and a half, 4 or 5 gb of space and 4 gb of memory (again, using the same computer I mentioned above).

[ccordoba12]

And the same number of cores (4).

[183amir]

conda-forge/opencv-feedstock#18 (comment)

[jankatins]

BTW, at least on windows it looks like jpeg 8 runtime dependencies are not needed as jpeg 8 only includes lib files (not dlls and implib), so the jpeg stuff is statically compiled into the package.

C:\portabel\miniconda\pkgs\pyqt-4.11.4-py35_5\Lib\site-packages\PyQt4
λ dumpbin.exe /DEPENDENTS *pyd |grep jpeg

-> It seems that the defaults packages, at least on windows, does not need to depend on jpeg!

[Also posted here: https://github.com/conda-forge/libpng-feedstock/issues/7#issuecomment-224398187, which deals with splitting and renaming the jpeg package to make different versions coinstallable]

[patricksnape]

To clarify, I am happy to write a short email to both CircleCI and Travis asking for 2 hour build timeouts and 5GB of storage. I doubt we will get more cores/RAM, however. Maybe if we tried the build and then we have an example to point to showing them what we are seeing?

As a note, on the Menpo team we ended up switching to our own Jenkins infrastructure hosted internally for both OSX and we are transitioning for Windows. So I can understand why we would want to switch. However, the biggest downside is that your forget how much compute power is behind services such as Travis and it may well end up being slower in the worst case.

However, if we do switch to a Jenkins-like infrastructure I am happy to petition my department to add some cloud compute Linux instances to contribute as slaves.

[ocefpaf]

@patricksnape what is your timezone? I would love if you could login in our next meeting and talk about that with @kwilcox who has some similar ideas.

PS: still thing that a short e-mail for CircleCI and Travis-CI is worth it. Thanks for volunteering to do so.

[patricksnape]

UTC +1 - When is the call?

[ocefpaf]

I am having trouble connecting to the hackpad right now, but the meeting info should be here:

https://conda-forge.hackpad.com/conda-forge-meetings-2YkV96cvxPG#:h=2016-05-13

I think it is 13:00 UTC.

[jakirkham]

You mean 1400UTC, right? 😉

[hadim]

Did someone send a mail to CircleCI and Travis-CI guys at then end ? If you didn't are you still looking to build your own CI infrastructure ?

(it's such a shame I can't use opencv and matplotlib in the same conda env)

[patricksnape]

I believe I did send one to the Travis but I can't seem to find the response anywhere - perhaps I didn't get one (or I'm misremembering). I think that building our own CI infrastructure would be pretty difficult for non-Linux builds because OSX and Windows can't be used with docker. I think the primary issue is that, with the volume of builds that we have, we need a fairly serious coordination across many institutions to enable timely builds. Either that or we need sponsorship for paid CircleCI and travis builds so that we can more easily request increased build capabilities.

I think we can revisit this - I just would need an example failing build for each platform (with the reasons for the failures so I know what resources to request) to use as reference when contacting the CI providers.

[jakirkham]

This issue is clearly not getting any traction. Also as Qt is now packaged on Linux some of the problems here have been resolved due to that. Am going to close this out. If others feel strongly about this change or have other issues, please raise them separately. Thanks.