Fix pin for OS X #211

ocefpaf · 2016-08-17T15:30:31Z

In light of the test performed in conda-forge/matplotlib-feedstock#52 (comment) I can confirm that conda-forge/matplotlib-feedstock#41 happens only on OS X.

This closes conda-forge/matplotlib-feedstock#41

patricksnape · 2016-08-18T12:58:56Z

This pinning is fine by me.

pelson · 2016-08-19T21:32:12Z

Me too. I'm happy for somebody else to merge this. Once done, I will re-create the conda-forge-maint Heroku image and we will start to see the recent pinnings get rolled out.

jakirkham · 2016-08-19T21:55:05Z

What's the down vote for, @183amir?

183amir · 2016-08-19T21:57:47Z

Because when libpng developers break API in a patch version, what stops them from braking it again in the future? If we go ahead with this, we might need to remove/hotfix some packages in future.

pelson · 2016-08-21T07:23:12Z

we might need to remove/hotfix some packages in future.

I understand your reluctance @183amir, but the same statement can be made for all of our pinnings. I completely accept that it has been particularly acute for libpng though. Has anybody tried contacting the developers to find out what their versioning scheme actually is? There is no point us trying to reverse engineer this - I'm sure they will be able to tell us in a heartbeat what the intended backwards compatibility is.

ocefpaf · 2016-08-21T12:58:08Z

Has anybody tried contacting the developers to find out what their versioning scheme actually is?

I did. Never got an answer... I was curious why we see the breakages from 1.6.21 to 1.2.23 only on OS X.

jakirkham · 2016-08-21T15:29:16Z

This is interesting. According to this ABI compatibility chart, there really shouldn't be these breaks. If this is true, what is really going on? In particular, what about Mac doesn't follow this? It may be worth digging into compilation details to answer this.

jakirkham · 2016-08-23T00:37:53Z

scripts/pin_the_slow_way.py

@@ -41,7 +41,7 @@
          'libblitz': 'libblitz 0.10|0.10.*',
          'libmatio': 'libmatio 1.5.*',
          'libnetcdf': 'libnetcdf 4.4.*',
-          'libpng': 'libpng >=1.6.21,<1.7',
+          'libpng': 'libpng >=1.6.23,<1.7',


According to this chart, this should be 22.

pelson · 2016-08-23T07:30:30Z

I did. Never got an answer...

That is a shame. What medium did you use? We should try another route.

ocefpaf · 2016-08-23T11:30:50Z

That is a shame. What medium did you use?

To be fair it was not me. It was colleague that is writing the re-pinning scripts for fetched recipes. BTW he e-mailed them.

We should try another route.

There are mailing lists we can try.

We are happy with the pinning here and we know that works for our local channel. I sent it to conda-forge as a compromise between @183amir's proposal to pin down to the patch level and other people who are relying on tables that are failing us. (The backward compatibility there is 100% on many version we know are failing.) I don't really have a stake on this.

183amir · 2016-09-27T19:29:44Z

The API/ABI is again broken between 1.6.23 and 1.6.24 on Mac. Reported here first: conda-forge/libpng-feedstock#10

jakirkham · 2016-09-27T19:42:23Z

I really think we should revisit what @JanSchulz told us many months ago and wrote up in issue ( conda-forge/libpng-feedstock#7 ). IMHO what he wrote should be rolled into a CFEP and discussed there.

The pinning system (while less broken than no pinnings) is still broken. Continuing with it is resulting debates/cognitive load w.r.t. hot fixing, deletions, otherwise broken old packages, changing pinnings, conda-build enhancements, automation, and having multiple SONAMEs installed simultaneously. If we just use the SONAME, these problems are basically gone. There may be some cross OS tricks, but that would be worth looking at more closely.

If someone(s) want to put this into a CFEP PR, we would be better served by reviewing that change and updating how we do things in conda-forge.

ocefpaf · 2016-09-27T21:31:32Z

I did not see any issues mixing libpng 1.6.23 and 1.6.24 in #211 and the tests in conda-forge/matplotlib-feedstock#52 (comment)

Either way I no longer have a stake on this as I re-pin them for our local builds. BTW the vast majority of our users are OS X guys and they never report any issue with the pinning in #211

pelson · 2016-09-30T05:38:42Z

Either way I no longer have a stake on this as I re-pin them for our local builds. BTW the vast majority of our users are OS X guys and they never report any issue with the pinning in #211

Thanks @ocefpaf. Useful to know you are re-pinning. In general, do you have an aspiration to have the pinning as similar as possible to conda-forge or Continuum, or does it simply not matter?

If anybody is able to give a concrete understanding of what is going on in conda-forge/libpng-feedstock#10 I think we should really have that written down. Otherwise, I'm afraid we are going to have to pin libpng to the "patch" version (where major.minor.patch in SymVer).

ocefpaf · 2016-09-30T13:13:02Z

Thanks @ocefpaf https://github.com/ocefpaf. Useful to know you are
re-pinning. In general, do you have an aspiration to have the pinning as
similar as possible to conda-forge or Continuum, or does it simply not
matter?

Closer to Continuum only b/c of the lack of a Qt build for Windows on our
side. I believe that will be fixed soon (thanks to Mike
conda/conda-build#1413) and the pinnings will be
the same as conda-forge.

If anybody is able to give a concrete understanding of what is going on in
conda-forge/libpng-feedstock#10
conda-forge/libpng-feedstock#10 I think we
should really have that written down. Otherwise, I'm afraid we are going to
have to pin libpng to the "patch" version (where major.minor.patch in
SymVer).

The tests I performed some time ago did not should that issue on OS X but
apparently they re-surfaced. I also never saw that issue on Linux nor
Windows. B/c it is quite complicated to understand what is going on with OS
X using only Travis-CI so I am OK with a patch level pinning.

ocefpaf · 2016-09-30T13:39:50Z

scripts/pin_the_slow_way.py

@@ -39,7 +39,7 @@
          'libblitz': 'libblitz 0.10|0.10.*',
          'libmatio': 'libmatio 1.5.*',
          'libnetcdf': 'libnetcdf 4.4.*',
-          'libpng': 'libpng >=1.6.21,<1.7',
+          'libpng': 'libpng 1.6.24',


Since I am leaving in a few hours I added this commit to easy @183amir's mind.

183amir · 2016-10-01T16:06:20Z

Proof that matplotlib compiled with 1.6.24 does not work with 1.6.23 in travis:

conda-forge/matplotlib-feedstock#64

$ conda create -n temp01 --override-channels -c conda-forge -c defaults --use-local python=2.7 matplotlib --yes

Using Anaconda API: https://api.anaconda.org

Fetching package metadata ...........

Solving package specifications: ..........

Package plan for installation in environment /Users/travis/miniconda3/envs/temp01:

The following packages will be downloaded:

    package                    |            build

    ---------------------------|-----------------

    mkl-11.3.3                 |                0       101.4 MB  defaults

    numpy-1.10.4               |           py27_2         3.0 MB  defaults

    ------------------------------------------------------------

                                           Total:       104.5 MB

The following NEW packages will be INSTALLED:

    ca-certificates: 2016.8.31-0       conda-forge

    certifi:         2016.8.31-py27_0  conda-forge

    cycler:          0.10.0-py27_0     conda-forge

    freetype:        2.6.3-1           conda-forge

    functools32:     3.2.3.2-py27_1    conda-forge

    libpng:          1.6.24-0          conda-forge

    matplotlib:      1.5.3-np110py27_1 local      

    mkl:             11.3.3-0          defaults   

    ncurses:         5.9-9             conda-forge

    numpy:           1.10.4-py27_2     defaults   

    openssl:         1.0.2h-2          conda-forge

    pyparsing:       2.1.8-py27_0      conda-forge

    python:          2.7.12-1          conda-forge

    python-dateutil: 2.5.3-py27_0      conda-forge

    pytz:            2016.6.1-py27_0   conda-forge

    readline:        6.2-0             conda-forge

    setuptools:      26.1.1-py27_0     conda-forge

    six:             1.10.0-py27_0     conda-forge

    sqlite:          3.13.0-1          conda-forge

    tk:              8.5.19-0          conda-forge

    zlib:            1.2.8-3           conda-forge

Fetching packages ...

mkl-11.3.3-0.t 100% || Time: 0:00:05  18.01 MB/s

numpy-1.10.4-p 100% || Time: 0:00:00   8.45 MB/s

Extracting packages ...

[      COMPLETE      ]|| 100%

Linking packages ...

[      COMPLETE      ]|| 100%

No psutil available.

To proceed, please conda install psutil#

# To activate this environment, use:

# $ source activate temp01

#

# To deactivate this environment, use:

# $ source deactivate

#

The command "conda create -n temp01 --override-channels -c conda-forge -c defaults --use-local python=2.7 matplotlib --yes" exited with 0.

1.00s$ source activate temp01

The command "source activate temp01" exited with 0.

2.00s$ python -c 'import matplotlib.pyplot'

/Users/travis/miniconda3/envs/temp01/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.

  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

The command "python -c 'import matplotlib.pyplot'" exited with 0.

7.00s$ conda install --no-update-deps libpng=1.6.23 --no-channel-priority --yes

Using Anaconda API: https://api.anaconda.org

Fetching package metadata .........

Solving package specifications: ..........

Package plan for installation in environment /Users/travis/miniconda3/envs/temp01:

The following packages will be downloaded:

    package                    |            build

    ---------------------------|-----------------

    libpng-1.6.23              |                0         319 KB  conda-forge

The following packages will be DOWNGRADED due to dependency conflicts:

    libpng: 1.6.24-0 conda-forge --> 1.6.23-0 conda-forge

Fetching packages ...

libpng-1.6.23- 100% || Time: 0:00:00   3.20 MB/s

Extracting packages ...

[      COMPLETE      ]|| 100%

Unlinking packages ...

[      COMPLETE      ]|| 100%

Linking packages ...

[      COMPLETE      ]|| 100%

The command "conda install --no-update-deps libpng=1.6.23 --no-channel-priority --yes" exited with 0.

3.00s$ python -c 'import matplotlib.pyplot'

/Users/travis/miniconda3/envs/temp01/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.

  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/Users/travis/miniconda3/envs/temp01/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>

    import matplotlib.colorbar

  File "/Users/travis/miniconda3/envs/temp01/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>

    import matplotlib.collections as collections

  File "/Users/travis/miniconda3/envs/temp01/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>

    import matplotlib.backend_bases as backend_bases

  File "/Users/travis/miniconda3/envs/temp01/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 62, in <module>

    import matplotlib.textpath as textpath

  File "/Users/travis/miniconda3/envs/temp01/lib/python2.7/site-packages/matplotlib/textpath.py", line 18, in <module>

    from matplotlib.mathtext import MathTextParser

  File "/Users/travis/miniconda3/envs/temp01/lib/python2.7/site-packages/matplotlib/mathtext.py", line 60, in <module>

    import matplotlib._png as _png

ImportError: dlopen(/Users/travis/miniconda3/envs/temp01/lib/python2.7/site-packages/matplotlib/_png.so, 2): Library not loaded: @rpath/libpng16.16.dylib

  Referenced from: /Users/travis/miniconda3/envs/temp01/lib/python2.7/site-packages/matplotlib/_png.so

  Reason: Incompatible library version: _png.so requires version 41.0.0 or later, but libpng16.16.dylib provides version 40.0.0

183amir · 2016-10-01T16:09:07Z

If we are going to patch on a patch version which we should, it's better to pin to libpng 1.6.22 as long as the defaults channel uses that for anaconda.

183amir · 2016-10-01T17:02:53Z

This is not specific to matplotlib: the same thing on bob.io.image: conda-forge/bob.io.image-feedstock#19

ocefpaf · 2016-11-14T15:42:48Z

This was only to satisfy @183amir. Our builds are actually consistent and the issues only appear when a bad channel mix happens. So closing this.

jakirkham reviewed Aug 23, 2016
View reviewed changes

ocefpaf mentioned this pull request Sep 30, 2016

Update pinnings #175

Merged

ocefpaf force-pushed the libpng_pin branch from b850c38 to 55b61c2 Compare September 30, 2016 13:39

ocefpaf commented Sep 30, 2016

View reviewed changes

ocefpaf added 2 commits September 30, 2016 10:41

Fix pin for OS X

ebdc693

Pin to patch level

55b61c2

ocefpaf closed this Nov 14, 2016

ocefpaf deleted the libpng_pin branch November 14, 2016 15:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix pin for OS X #211

Fix pin for OS X #211

ocefpaf commented Aug 17, 2016

patricksnape commented Aug 18, 2016

pelson commented Aug 19, 2016

jakirkham commented Aug 19, 2016

183amir commented Aug 19, 2016

pelson commented Aug 21, 2016

ocefpaf commented Aug 21, 2016

jakirkham commented Aug 21, 2016

jakirkham Aug 23, 2016

pelson commented Aug 23, 2016

ocefpaf commented Aug 23, 2016

183amir commented Sep 27, 2016

jakirkham commented Sep 27, 2016

ocefpaf commented Sep 27, 2016

pelson commented Sep 30, 2016

ocefpaf commented Sep 30, 2016

ocefpaf Sep 30, 2016

183amir Sep 30, 2016

183amir commented Oct 1, 2016

183amir commented Oct 1, 2016

183amir commented Oct 1, 2016

ocefpaf commented Nov 14, 2016

Fix pin for OS X #211

Fix pin for OS X #211

Conversation

ocefpaf commented Aug 17, 2016

patricksnape commented Aug 18, 2016

pelson commented Aug 19, 2016

jakirkham commented Aug 19, 2016

183amir commented Aug 19, 2016

pelson commented Aug 21, 2016

ocefpaf commented Aug 21, 2016

jakirkham commented Aug 21, 2016

jakirkham Aug 23, 2016

Choose a reason for hiding this comment

pelson commented Aug 23, 2016

ocefpaf commented Aug 23, 2016

183amir commented Sep 27, 2016

jakirkham commented Sep 27, 2016

ocefpaf commented Sep 27, 2016

pelson commented Sep 30, 2016

ocefpaf commented Sep 30, 2016

ocefpaf Sep 30, 2016

Choose a reason for hiding this comment

183amir Sep 30, 2016

Choose a reason for hiding this comment

183amir commented Oct 1, 2016

183amir commented Oct 1, 2016

183amir commented Oct 1, 2016

ocefpaf commented Nov 14, 2016