Update dependency pinnings to latest versions and enable circleci

[johanneskoester]

Further, this PR removes the Python pinning, such that we can build for all supported version of python 3

[mbargull]

Could be that conda-forge/conda-feedstock#36 is only waiting on Travis CI..

[johanneskoester]

Indeed.

[mbargull]

This will automatically cause Python 3.6 to be used for the base build system -- not a bad thing, just an FYI.

[johanneskoester]

Yes, I have discovered this as well. I think that should be fine.

[johanneskoester]

I will make all those test cases that fail now due to 3.6 agnostic of the python version. Just waiting for all tests to finish first.

[johanneskoester]

Actually, I think this won't use 3.6 automatically. Instead, installing the package should use whatever is the default python version in the given conda installation.

[mbargull]

Correct, but since we use Miniconda3-4.3.21 we have Python 3.6 😉.

[johanneskoester]

Ok, so somebody must have updated it. Originally, I had set it to the latest Py35 version such that no new python has to be installed when setting up travis jobs. Good catch. One more reason to make bioconda-utils flexible in this regard.

[mbargull]

Ok, so somebody must have updated it

That would be me in #204, due to noarch: python issues.

[mbargull]

So, prior to #204 we had
defaults::conda=4.2.12/defaults::python=3.5
being updated to
conda-forge::conda=4.3.21/conda-forge::python=3.5.

Currently we have
defaults::conda=4.3.21/defaults::python=3.6
being updated/downgraded to
conda-forge::conda=4.3.21/conda-forge::python=3.5.

After this PR we would have
defaults::conda=4.3.21/defaults::python=3.6
being updated to
conda-forge::conda=4.3.31/conda-forge::python=3.6.

So overall, since we prioritize conda-forge over defaults, the amount of packages (including all dependencies) that get updated/replaced is more or less equal.

[johanneskoester]

Ah, of course the channel priorities :-). You are right, it does not really make a difference.

[mbargull]

At some point in the future we could be using some conda-forge installer. @jakirkham has done some initial work on this at https://github.com/jakirkham/miniforge (I still haven't come to check this out, yet!). Or we could use a Bioconda-specific one which builds on top of that (that would be helpful to cut down the OSX build/setup times -- for Linux we can always try to use a container-only workflow at some point).
But that needn't be something to do in the near future, though, as there are still a couple of rough edges in and around constructor that need to be smoothed out..

As for this PR:
You could also append .* to the jinja2 and pygithub versions (unless there are compatibility reasons not to do this). conda can install, e.g., a jinja2 2.10 given jinja2=2.10 or jinja2=2.10.*.

[jakirkham]

Thanks for the ping @mbargull. Yeah I hope that installer will be useful for speeding up CIs. Would be interested in any feedback any of you may have. Ultimately will want to move it to an org. Have tried out some version of them on all OSes.

There is some speculation as to whether it (or something like it) could be a community installer. Though it is a bit controversial. Would point to the thread following this comment for more details.

[johanneskoester]

For speeding up the CI, we could also make use of travis caching mechanisms. Seems to me this would be equivalent to using an installer. Am I missing something?

[jakirkham]

When using the macOS 10.10 image, caching incurs 3-5mins of setup time in my experience. That occurs even if the cache is empty. Have seen it go upwards of that as well. My last build using the cache on miniforge used ~5mins to setup the cache. Installing and upgrading Miniconda from scratch takes ~2mins. So caching is actually slightly worse than just using Miniconda and upgrading.

Have pressed Travis CI repeatedly to rebuild the image to avoid this overhead, but they don't appear to be interested. Issue ( travis-ci/travis-ci#9027 ) is relevant here.

Caching still runs into the same download count issues that another installer would except that we don't have a good alternative metric for how often we have restored the installer from cache. So there isn't a good way to return that info to Anaconda. IOW even if the time improved, there is a downside to this approach vs. building an installer and using it.

[johanneskoester]

@daler @bgruening I am unsure who of you added the those secret variables:

ENCRYPTED_KEY_VAR="encrypted_${ENCRYPTION_LABEL}_key"
ENCRYPTED_IV_VAR="encrypted_${ENCRYPTION_LABEL}_iv"
ENCRYPTED_KEY=${!ENCRYPTED_KEY_VAR}
ENCRYPTED_IV=${!ENCRYPTED_IV_VAR}

to the build-docs.sh script. They are defined via the Travis CI interface. As I am unable to migrate them to CircleCI, would you mind putting them here?

[johanneskoester]

@daler, @bgruening, we also need ANACONDA_TOKEN here.

[daler]

@johanneskoester I had added that, that's for the encrypted private key generated by travis-ci CLI (key.enc for pushing docs. The encryption label is https://github.com/bioconda/bioconda-utils/blob/master/.travis.yml#L16, but since it was originally encrypted using the travis-ci CLI I will have to make changes for circle-ci.

Looks to be mostly similar to the travis-ci setup, https://github.com/circleci/encrypted-files. I will get to this later today.

[johanneskoester]

Never mind, I am working on this already. Almost done!

[daler]

Thanks! Sorry I've been so busy lately, I've been following all your work though.

[johanneskoester]

@daler actually using custom SSH keys seems pretty easy in circleci compared to what we had before. Very convenient.

[daler]

Oh, that is really convenient!

[johanneskoester]

The only previously failing test case passes now. Hence, I will merge this now fast, for reasons outlined in our @bioconda/core gitter chat.