Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moar Psycopg2 #1162

Merged
merged 21 commits into from
Aug 1, 2016
Merged

Moar Psycopg2 #1162

merged 21 commits into from
Aug 1, 2016

Conversation

msarahan
Copy link
Member

Continuation of #291 - this adds the "libpq" package which is intended to be a repackaging of a minimal subset of postgresql - whatever psycopg2 needs at runtime.

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/libpq, recipes/psycopg2) and found some lint.

Here's what I've got...

For recipes/libpq:

  • The recipe must have a build/number section.

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/libpq, recipes/psycopg2) and found it was in an excellent condition.

@msarahan
Copy link
Member Author

This appears to be blocked on conda-build. I am resolving the issue, and will tag a new release soon.

@msarahan
Copy link
Member Author

All good here (sans win) when the new conda-build is released.

@msarahan
Copy link
Member Author

msarahan commented Aug 1, 2016

Conda-build 1.21.8 tagged. Available on Conda-Forge as of #1168. Let's see what happens...

@msarahan
Copy link
Member Author

msarahan commented Aug 1, 2016

Bad news with 1.21.8 creating a logger at https://github.com/conda/conda-build/blob/master/conda_build/__init__.py#L26 - we have some major issues with excessive output from conda, and I need to figure out better ways to have the logger for conda-build, but not interfere with logging decisions over in conda-land.

@msarahan
Copy link
Member Author

msarahan commented Aug 1, 2016

Should be all green here @ocefpaf, since circleCI was the only holdup after 1.21.8 - please take one more look and merge at your convenience.

@ocefpaf
Copy link
Member

ocefpaf commented Aug 1, 2016

Awesome @msarahan! Thanks!! I feel bad for all the trouble you went to finish this one.

build:
number: 0
skip: True # [win]
# features:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are going to try Windows right after feedstocking we can leave these here. No need to remove them now.

Copy link
Member Author

@msarahan msarahan Aug 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that was my hope. I don't know how quickly I'll get to it - still fighting postgresql build on win.

@ocefpaf ocefpaf merged commit f44ffe6 into conda-forge:master Aug 1, 2016
@msarahan msarahan deleted the psycopg2 branch August 1, 2016 22:13
@msarahan
Copy link
Member Author

msarahan commented Aug 1, 2016

Hooray! Thanks @ocefpaf and @jseabold!

@jiffyclub
Copy link

This seems to still not work for me with the same error as the original:

Creating a new environment:

~ > conda create -n pg2 -c conda-forge python=3.5 ipython psycopg2
Fetching package metadata .........
Solving package specifications: ..........

Package plan for installation in environment /Users/mattdavis/miniconda3/envs/pg2:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2016.2.28  |                0         157 KB  conda-forge
    ncurses-5.9                |                8        1000 KB  conda-forge
    sqlite-3.13.0              |                1         1.4 MB  conda-forge
    tk-8.5.19                  |                0         2.0 MB  conda-forge
    xz-5.2.2                   |                0         288 KB  conda-forge
    zlib-1.2.8                 |                3          90 KB  conda-forge
    openssl-1.0.2h             |                2         3.2 MB  conda-forge
    readline-6.2               |                0         311 KB  conda-forge
    python-3.5.2               |                2        24.7 MB  conda-forge
    certifi-2016.2.28          |           py35_1         211 KB  conda-forge
    decorator-4.0.10           |           py35_0          12 KB  conda-forge
    pickleshare-0.7.3          |           py35_0           9 KB  conda-forge
    ptyprocess-0.5.1           |           py35_0          19 KB  conda-forge
    pygments-2.1.3             |           py35_0         1.2 MB  conda-forge
    simplegeneric-0.8.1        |           py35_0           7 KB  conda-forge
    six-1.10.0                 |           py35_0          18 KB  conda-forge
    wcwidth-0.1.7              |           py35_0          22 KB  conda-forge
    pexpect-4.2.0              |           py35_1          67 KB  conda-forge
    prompt_toolkit-1.0.3       |           py35_0         307 KB  conda-forge
    setuptools-23.0.0          |           py35_1         461 KB  conda-forge
    traitlets-4.2.2            |           py35_0         116 KB  conda-forge
    ipython-5.0.0              |           py35_0        1008 KB  conda-forge
    wheel-0.29.0               |           py35_0          82 KB  conda-forge
    pip-8.1.2                  |           py35_0         1.6 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        38.2 MB

The following NEW packages will be INSTALLED:

    appnope:                            0.1.0-py35_0
    backports.shutil_get_terminal_size: 1.0.0-py35_0     conda-forge
    ca-certificates:                    2016.2.28-0      conda-forge
    certifi:                            2016.2.28-py35_1 conda-forge
    decorator:                          4.0.10-py35_0    conda-forge
    ipython:                            5.0.0-py35_0     conda-forge
    ipython_genutils:                   0.1.0-py35_0
    libpq:                              9.5.3-0          conda-forge
    ncurses:                            5.9-8            conda-forge
    openssl:                            1.0.2h-2         conda-forge
    pexpect:                            4.2.0-py35_1     conda-forge
    pickleshare:                        0.7.3-py35_0     conda-forge
    pip:                                8.1.2-py35_0     conda-forge
    prompt_toolkit:                     1.0.3-py35_0     conda-forge
    psycopg2:                           2.6.1-py35_0     conda-forge
    ptyprocess:                         0.5.1-py35_0     conda-forge
    pygments:                           2.1.3-py35_0     conda-forge
    python:                             3.5.2-2          conda-forge
    python.app:                         1.2-py35_4
    readline:                           6.2-0            conda-forge
    setuptools:                         23.0.0-py35_1    conda-forge
    simplegeneric:                      0.8.1-py35_0     conda-forge
    six:                                1.10.0-py35_0    conda-forge
    sqlite:                             3.13.0-1         conda-forge
    tk:                                 8.5.19-0         conda-forge
    traitlets:                          4.2.2-py35_0     conda-forge
    wcwidth:                            0.1.7-py35_0     conda-forge
    wheel:                              0.29.0-py35_0    conda-forge
    xz:                                 5.2.2-0          conda-forge
    zlib:                               1.2.8-3          conda-forge

Proceed ([y]/n)?

Pruning fetched packages from the cache ...
Fetching packages ...
ca-certificate 100% |#######################################################################################| Time: 0:00:00 497.22 kB/s
ncurses-5.9-8. 100% |#######################################################################################| Time: 0:00:00   1.46 MB/s
sqlite-3.13.0- 100% |#######################################################################################| Time: 0:00:00   1.77 MB/s
tk-8.5.19-0.ta 100% |#######################################################################################| Time: 0:00:00   2.47 MB/s
xz-5.2.2-0.tar 100% |#######################################################################################| Time: 0:00:00 621.35 kB/s
zlib-1.2.8-3.t 100% |#######################################################################################| Time: 0:00:00 384.53 kB/s
openssl-1.0.2h 100% |#######################################################################################| Time: 0:00:01   3.19 MB/s
readline-6.2-0 100% |#######################################################################################| Time: 0:00:01 252.62 kB/s
python-3.5.2-2 100% |#######################################################################################| Time: 0:00:04   6.19 MB/s
certifi-2016.2 100% |#######################################################################################| Time: 0:00:01 215.18 kB/s
decorator-4.0. 100% |#######################################################################################| Time: 0:00:00  13.27 MB/s
pickleshare-0. 100% |#######################################################################################| Time: 0:00:00   9.00 MB/s
ptyprocess-0.5 100% |#######################################################################################| Time: 0:00:00 256.21 kB/s
pygments-2.1.3 100% |#######################################################################################| Time: 0:00:00   1.51 MB/s
simplegeneric- 100% |#######################################################################################| Time: 0:00:00 632.06 kB/s
six-1.10.0-py3 100% |#######################################################################################| Time: 0:00:00 241.34 kB/s
wcwidth-0.1.7- 100% |#######################################################################################| Time: 0:00:00 283.57 kB/s
pexpect-4.2.0- 100% |#######################################################################################| Time: 0:00:00 300.73 kB/s
prompt_toolkit 100% |#######################################################################################| Time: 0:00:00 818.59 kB/s
setuptools-23. 100% |#######################################################################################| Time: 0:00:00 960.48 kB/s
traitlets-4.2. 100% |#######################################################################################| Time: 0:00:00 438.35 kB/s
ipython-5.0.0- 100% |#######################################################################################| Time: 0:00:00   1.82 MB/s
wheel-0.29.0-p 100% |#######################################################################################| Time: 0:00:00 219.75 kB/s
pip-8.1.2-py35 100% |#######################################################################################| Time: 0:00:00   2.29 MB/s
Extracting packages ...
[      COMPLETE      ]|##########################################################################################################| 100%
Linking packages ...
[      COMPLETE      ]|##########################################################################################################| 100%
No psutil available.
To proceed, please conda install psutil#
# To activate this environment, use:
# $ source activate pg2
#
# To deactivate this environment, use:
# $ source deactivate
#

Trying to connect with sslmode='require':

~ > ca pg2
(pg2) ~ > ipython
Python 3.5.2 | packaged by conda-forge | (default, Jul 26 2016, 01:37:38)
Type "copyright", "credits" or "license" for more information.

IPython 5.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import psycopg2

In [2]: psycopg2.connect(user='mattdavis', host='localhost', port=5432, database='postgres', sslmode='require')
---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
<ipython-input-2-7c777b9d2035> in <module>()
----> 1 psycopg2.connect(user='mattdavis', host='localhost', port=5432, database='postgres', sslmode='require')

/Users/mattdavis/miniconda3/envs/pg2/lib/python3.5/site-packages/psycopg2/__init__.py in connect(dsn, database, user, password, host, port, connection_factory, cursor_factory, async, **kwargs)
    162                 for (k, v) in items])
    163
--> 164     conn = _connect(dsn, connection_factory=connection_factory, async=async)
    165     if cursor_factory is not None:
    166         conn.cursor_factory = cursor_factory

OperationalError: sslmode value "require" invalid when SSL support is not compiled in

conda list shows everything installed from conda-forge:

(pg2) ~ > conda list
# packages in environment at /Users/mattdavis/miniconda3/envs/pg2:
#
appnope                   0.1.0                    py35_0
backports.shutil_get_terminal_size 1.0.0                    py35_0    conda-forge
ca-certificates           2016.2.28                     0    conda-forge
certifi                   2016.2.28                py35_1    conda-forge
decorator                 4.0.10                   py35_0    conda-forge
ipython                   5.0.0                    py35_0    conda-forge
libpq                     9.5.3                         0    conda-forge
ncurses                   5.9                           8    conda-forge
openssl                   1.0.2h                        2    conda-forge
pexpect                   4.2.0                    py35_1    conda-forge
pickleshare               0.7.3                    py35_0    conda-forge
pip                       8.1.2                    py35_0    conda-forge
prompt_toolkit            1.0.3                    py35_0    conda-forge
psycopg2                  2.6.1                    py35_0    conda-forge
ptyprocess                0.5.1                    py35_0    conda-forge
pygments                  2.1.3                    py35_0    conda-forge
python                    3.5.2                         2    conda-forge
readline                  6.2                           0    conda-forge
setuptools                23.0.0                   py35_1    conda-forge
simplegeneric             0.8.1                    py35_0    conda-forge
six                       1.10.0                   py35_0    conda-forge
sqlite                    3.13.0                        1    conda-forge
tk                        8.5.19                        0    conda-forge
traitlets                 4.2.2                    py35_0    conda-forge
wcwidth                   0.1.7                    py35_0    conda-forge
wheel                     0.29.0                   py35_0    conda-forge
xz                        5.2.2                         0    conda-forge
zlib                      1.2.8                         3    conda-forge
ipython_genutils          0.1.0                    py35_0
python.app                1.2                      py35_4

conda info:

(pg2) ~ > conda info
Current conda install:

             platform : osx-64
        conda version : 4.1.11
    conda-env version : 2.5.2
  conda-build version : not installed
       python version : 3.5.2.final.0
     requests version : 2.10.0
     root environment : /Users/mattdavis/miniconda3  (writable)
  default environment : /Users/mattdavis/miniconda3/envs/pg2
     envs directories : /Users/mattdavis/miniconda3/envs
        package cache : /Users/mattdavis/miniconda3/pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/osx-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/osx-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
         offline mode : False
    is foreign system : False

(pg2) ~ >

@msarahan
Copy link
Member Author

msarahan commented Aug 2, 2016

Yep, sorry, this appears to be disabled upstream in the postgresql feedstock: https://github.com/conda-forge/postgresql-feedstock/blob/master/recipe/build.sh#L6-L7

We'll look into it.

@jseabold
Copy link
Contributor

jseabold commented Aug 2, 2016

Can you explain that a bit more? He doesn't appear to have installed the conda-forge postgresql? If I have a system postgresql with proper ssl support (like the Postgres.App), should psycopg2 work?

@msarahan
Copy link
Member Author

msarahan commented Aug 2, 2016

The conda-forge psycopg2 depends on libpq - which is a rebundle of our postgresql build's libpq and pg-config. If you manually delete those, and your system postgresql gets found, then it might work.

@jseabold
Copy link
Contributor

jseabold commented Aug 2, 2016

Right. Is there any discussion of that code comment above? I might be able to look in to this sometime this week.

Might it be worth not listing libpq as a dependency and letting users take care of this? Or better yet to register an external dependency? Something like update-alternatives?

@msarahan
Copy link
Member Author

msarahan commented Aug 3, 2016

Right now, listing libpq is the best we can do. We need to cater to the lowest common denominator. Something like update-alternatives would be great, but that's a conda development project. Many people internally are interested in such a thing (also discussed as the concept of "provides"), but I don't know where it is on the road map.

@msarahan
Copy link
Member Author

@jiffyclub, this should be fully functional as of conda-forge/libpq-feedstock#1

@jiffyclub
Copy link

Thank you @msarahan, confirmed this on my end!

@msarahan
Copy link
Member Author

Excellent. Glad it works. I'm also working on defaults, but compilers are not being agreeable. It'll also be up when I can get the kinks worked out.

for LIBRARY in `find ${SP_DIR}/${PKG_NAME} -name "*.so"`;
do
install_name_tool -change libssl.1.0.0.dylib @rpath/libssl.1.0.0.dylib $LIBRARY
install_name_tool -change libcrypto.1.0.0.dylib @rpath/libcrypto.1.0.0.dylib $LIBRARY
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Continuum would build openssl with conda-build this wouldn't be needed (c.f. ContinuumIO/anaconda-issues#498).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming these files are part of a different package (otherwise conda build would just do this itself after the build is complete). Assuming that is correct, shouldn't you break the hard links here? It doesn't matter for conda-forge, but if someone tries to build this recipe on their own machine, this will write into the package cache. Sorry if I'm missing something here.

Copy link
Member

@jakirkham jakirkham Aug 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't matter for conda-forge, but if someone tries to build this recipe on their own machine, this will write into the package cache.

👍 Absolutely correct. Should be a simple copy shuffle to break the hard links.

Could you please raise the issue on the feedstock though as this is already merged or better yet propose the change?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are they actually from a separate package? Based on the loop, it looks like they are part of this package, but if that's the case, why are these lines needed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

7 participants