Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom conda-forge distribution #90

Closed
danielfrg opened this issue Apr 14, 2016 · 29 comments
Closed

Custom conda-forge distribution #90

danielfrg opened this issue Apr 14, 2016 · 29 comments

Comments

@danielfrg
Copy link
Member

Moving some discussion from #22 to a separate issue.

With the new conda constructor its really easy to make a custom conda distribution (for all the platforms: OS X, Linux and Windows) with custom packages from a conda channel. I just tested it with a file like this:

name: centonda
version: 1.0.0

channels:
  - http://repo.continuum.io/pkgs/free/
  - https://conda.anaconda.org/conda-forge

specs:
  - python
  - conda
  - anyjson

At the moment you still need http://repo.continuum.io/pkgs/free/ in the channels list to have python and conda but you can see the idea, if these packages are on the conda-forge channel it would be possible to create a distribution with only community created packages.

It would also be possible to make that custom distribution point to the conda-forge channel by default. Not as straight forward but possible, see conda/constructor#16. With this is possible to keep people updated faster as the channel is updated.
@danielfrg danielfrg mentioned this issue Apr 14, 2016
Closed
@jjhelmus
Copy link
Contributor

@danielfrg I while back I used conda constructor to build a Miniconda like installer from home-build package that is self-hosting (the installer can be used to build itself). The recipes, scripts and such are in the acpd/acpd repository. Currently it only builds on a linux-64 platform but I have modification which seem to work on OS X that I will push after a bit more testing.

@jakirkham
Copy link
Member

Nice! Does this Mac support live in a branch somewhere?

@jjhelmus
Copy link
Contributor

I'd be interested to move this recipes over to conda-forge if the organization is open to having it packaging Python, etc that is already in the default conda channel. There are a bunch of open issues and PR on this topic that I could link but itstead I'll add an item to tomorrows conference call, #89.

@jjhelmus
Copy link
Contributor

Nice! Does this Mac support live in a branch somewhere?

Only on my machine, I will push it up to the main repo later today.

@jjhelmus jjhelmus mentioned this issue Apr 14, 2016
Closed
@jakirkham
Copy link
Member

I'd be interested to move this recipes over to conda-forge if the organization is open to having it packaging Python, etc that is already in the default conda channel.

I would be in favor of it. I know @pelson seems to be pretty interested. Though he has just been pretty busy keeping conda-forge running smoothly and automating things. @ocefpaf seems to be embracing the build the world philosophy. 😄

There are a bunch of open issues and PR on this topic that I could link but itstead I'll add an item to tomorrows conference call, #89.

Please do. ...or I guess you already did. 😄

@jakirkham
Copy link
Member

Only on my machine, I will push it up to the main repo later today.

Cool, I'd be curious to play with this.

@danielfrg
Copy link
Member Author

@jjhelmus That is great work. I think that the core packages recipes and a custom distribution would be a great addition to conda-forge and let people get started easier with the conda-forge packages.

In the future and maybe with some hosting conda-forge could also host a custom conda channel independent from anaconda.org. All the machinery is already in conda and I have a docker container to do that here: https://github.com/danielfrg/docker-conda-channel. All this is future plans but I hope this movement keeps going :)

@jakirkham
Copy link
Member

In the future and maybe with some hosting conda-forge could also host a custom conda channel independent from anaconda.org.

You're meaning another website with channels? Maybe, but I'm less clear what we are gaining from this. We benefit by having a close working relationship with Continuum.

@jjhelmus
Copy link
Contributor

@danielfrg Thanks. I like the idea of having a second source for hosting conda packages, Continuum does a great job with Anaconda.org but having a second option is never a bad idea.

I really want to get a windows miniconda clone working and conda constructor can create Windows installers. The blockage for me is getting Python 3.5 to compile on Windows. I now have a Windows machine just for building conda packages so hopefully I (or someone smarter than me!) can figure this one out.

@pelson
Copy link
Member

pelson commented Apr 15, 2016

if the organization is open to having it packaging Python

It has languished (through other priorities on conda-forge) but I already have a PR for python: conda-forge/staged-recipes#84 If you want to take that forwards 👍

@jjhelmus
Copy link
Contributor

The osx_build branch of the acpd repository has recipes for Python, etc that work on OS X, although these seems to be an issue with conda/conda-env that I need to look into. I still need to figure out the best manner to script the build on OS X before merging. I'll likely rebase that branch to condense some of the commits before merging.

I'll be submitting a Python 3.5 PR to conda-forge later today. I think starting fresh is a better option than trying to work off conda-forge/staged-recipes#84

@jakirkham
Copy link
Member

More food for thought on the hosting packages elsewhere ( http://wesmckinney.com/blog/the-problem-with-conda-forge-right-now/ ). One thing to consider along this line might be using something like Bintray as a secondary package host.

Still not sure that this is the biggest or most pressing problem facing our community at present, but I suppose it is worth considering down the road.

@pelson
Copy link
Member

pelson commented Apr 22, 2016

Still not sure that this is the biggest or most pressing problem facing our community at present

Completely agreed. It doesn't keep me awake at night like some of the other issues we need to deal with 💤 😉 .

@jakirkham jakirkham mentioned this issue Jul 16, 2016
Closed
31 tasks
This was referenced Oct 28, 2016
Closed
Open
@jakirkham
Copy link
Member

We have created a repo for working on the installer. There are a number of issues that we would love feedback and help working on over there. They include such things as simply naming the installer, versioning, packages to include, CI adjustments, and making releases. If you have time and interest, please take a look.

cc @conda-forge/core

@ChrisBarker-NOAA
Copy link
Contributor

ChrisBarker-NOAA commented Mar 16, 2017 via email

@jakirkham jakirkham mentioned this issue Sep 14, 2017
Open
@jakirkham
Copy link
Member

Have you seen issue ( conda-forge/conda-smithy#569 ), @ChrisBarker-NOAA?

@jakirkham
Copy link
Member

Hi all, recently put some work into making a constructor-based conda-forge installer using CIs to build, test, and release the installer as GitHub release assets. Have called it miniforge. Just recently generated installers for all Python versions, OSes, and architectures supported by conda-forge. Each installer also has a corresponding sha256 checksum for verification. Feel free to try out the installers. Would be interested any feedback you have over at that repo.

@scopatz
Copy link
Member

scopatz commented Jan 10, 2018

Amazing @jakirkham!

@synapticarbors
Copy link
Member

Just fyi - miniforge is the subject of a post on the anaconda mailing list (https://groups.google.com/a/continuum.io/d/msg/anaconda/nxcQ41YlYBA/DeAFD4mXAAAJ). I'm not entirely sure if it was suppose to be sent to the general community -- seems like something that could have been brought up directly with initial questions about the intended use of the project.

@msarahan
Copy link
Member

Sorry, that was supposed to be an internal post. Yes, we are concerned about the intended use of the project. As I mentioned in that comment, I do definitely see positive use for a "miniconda++" that bootstraps conda-forge more quickly. I also see positive use for a pure miniconda clone - having conda-forge be able to bootstrap itself. Unfortunately, that positive use of the latter may detract from Anaconda's funding metrics. If possible, I would strongly encourage creation of the former: a bootstrap, dev tool, not a user-facing tool.

@jakirkham
Copy link
Member

Hi Mike, thanks for the feedback. In the interest of complete honesty, I had read the original post too.

Have to say though, the response comes as bit of a surprise to me. Maybe it shouldn't have? I guess after the idea of having an installer has been discussed on conda-forge for almost two years both on GitHub and in conda-forge meetings (including work done at the 2016 SciPy hackaton) intentions laid bare (CI speedup) and particularly with encouragement and support provided by a few people at Anaconda, would have figured there would be a bit more celebration at having accomplished the biggest piece. Appears I am sorely mistaken.

Given there seems to be a grave misunderstanding, let me be totally clear. The intention of this installer is to be given to conda-forge for use on CIs. The main goal has always been CI speedup and reduced load on Anaconda for common assets (e.g. conda, conda-build, etc.). Though now I'm wondering if the latter is problematic if this load fits into your financial metrics somehow. Would add that it would be very foolish for anyone to think they are going to get the same level of support from this installer as they would get from an Anaconda provided installer.

In addition to speeding up CIs, would note there is a fairly large amount of technical debt that has accumulated in conda-forge because we don't use our own installer (e.g. conda-forge-build-setup, the initial addition of things like libedit, frequent re-renderings to fix other issues ( as a recent example conda-forge/conda-smithy#631 ), etc.). Having our own installer would curb this clutter as we would start out with something that is compatible with our own stack to begin with. This will also free up people's time to work on more interesting pursuits in conda-forge or wherever else their work occurs.

Ideally the installer would actually include more packages (e.g. conda-build, anaconda-client, etc.). IOW whatever a feedstock needs to build and upload a package. As right now it does not, it is not as useful as it can be to conda-forge. However the noarch issues with constructor block that ATM. Though we are discussing these things in other issues. So we need not discuss them in this thread too. Once they are resolved, would be happy to include a more sizable portion of the stack in the installer. In any event, including these other packages was always part of the plan.

As more of a general note, we are all very appreciative of the work that Anaconda Inc. does. It is a great asset to the scientific community. The installer's intent was only to speed up conda-forge's CIs and cutdown on redundant load to Anaconda Inc's servers, which seems like the least we can do. In this way would hope that it would be an asset for the community (Anaconda included). Am truly sorry for any misunderstandings its publication has caused.

@msarahan
Copy link
Member

Thanks John. I'm very happy to hear about the intent, and I'm sorry I inferred too much from the recipe and its current lack of other packages.

I don't know what the cost/benefit is of conda-forge using our download hosting vs. providing their own. I'll research that, and we should do whatever the best thing is mutually. I'm not sure how much github would welcome being a content provider for these installers, anyway, but that's up to them.

We have wanted to provide a "deviconda" type installer for some time now. Perhaps we can work together on that - either by moving the noarch progress forward with constructor, or your idea to have conda convert be able to convert noarch packages to arch-specific packages, for constructor to then consume.

Again, sorry, and thanks for clarifying.

@pelson
Copy link
Member

pelson commented Jan 11, 2018

Unfortunately, that positive use of the latter may detract from Anaconda's funding metrics

I (we at conda-forge) have no interest in detracting from Anaconda's funding metrics. Personally, I'd like to see conda-forge have the ability to stand on its own feet (should Anaconda disappear for some reason), but I'd hate for that to be at the cost of Anaconda disappearing (or even losing business).

In these situations it is always worth reviewing what the metrics are, and whether they are measuring the right thing. Taking the worst case scenario from an Anaconda perspective, if conda-forge were to create a "miniforge" then I see no reason we couldn't feed the numbers back into the metric (since conda-forge would be shipping conda, a product of Anaconda Inc.). I'm just saying, let's figure out what we want from a conda-forge perspective, and make sure we have sensible conversations with the appropriate folks at Anaconda Inc. to make these goals happen without impacting downstream metrics/funding.

Great work on producing miniforge @jakirkham. @msarahan, should conda-forge choose the most extreme option of producing a general purpose open conda installer (i.e. an open version of miniconda) who would be the best person to speak to to ensure that we can provide the appropriate data to avoid causing a metrics issue for Anaconda Inc.?

@msarahan
Copy link
Member

Thanks @pelson. The best person to talk to is either me or @kalefranz or both. Kale leads our CDN efforts, so he's often one of the best people to talk with about collecting or aggregating metrics. Either of us could then forward you on to the internal team that crunches those numbers, which is in flux right now.

@pelson
Copy link
Member

pelson commented Jan 11, 2018

OK. Let's make sure conda-forge doesn't organically produce a miniforge that is general purpose (user facing) without doing our due diligence with @kalefranz to see if we can try to get the numbers into Anaconda's metrics. I'm more than happy (to be but don't need) to be involved @jakirkham if you think miniforge is on the verge of being ready for prime time. However, if we want to just focus on improving our build times for now (using a "minibuildforge"), then we can delay that conversation a little (though we would still be impacting the number of downloads of Miniconda fairly substantially).

Work for all? 👍 / 👎

@msarahan
Copy link
Member

msarahan commented Jan 11, 2018 via email

@pelson
Copy link
Member

pelson commented Jan 11, 2018 via email

@jakirkham
Copy link
Member

Interesting. If you guys want to talk about miniforge being some sort of conda-forge installer hypothetically, I'm not against it. Just this was not my initial intent as already explained above. Would add that I don't plan on being a sole maintainer of any such installer. So this would require community engagement to be viable. Would be happy to move the code into a community org to facilitate this.

As far as getting download count, GitHub provides an API for releases, which includes the download count. Note that this is a public API that can be used even without having any permissions on the repo or needing a GitHub token.

Regarding GitHub being a provider of releases, GitHub's limitations are simply release size, which we are well within. That document explicitly states there is no bandwidth limit nor a total size limit over all releases. It wouldn't hurt to verify this as well, but it would only make sense to do once we have numbers to share with them. Maybe you were thinking of GitHub Pages, which does have a bandwidth limit. Though there would be no harm in us caching binaries with Cloudflare or periodically cleaning up old installers, these don't appear to be requirements.

AFAIC the installer is ready for business. Wouldn't have mentioned it here if I didn't think so. 😉 While there is certainly room for improvement on the installer, there is nothing blocking its usage on my end. In fact we include tests in CI that run the installer and try a few basic conda commands, which all pass. Would certainly appreciate it if a few people played with it before we trying to push its usage.

Would really appreciate your engagement in these discussions Phil (if you are willing). 😄

@jakirkham jakirkham mentioned this issue Jan 17, 2018
Merged
@SylvainCorlay
Copy link
Member

Regarding a general-purpose miniforge, I think that it is becoming a requirement for binary compatibility purpose.

At the moment, if r is installed from conda-forge, we end up with packages from default whenever we e.g. install cmake (also from conda-forge).

As a package author, I would use a pure conda-forge environment should there be one available.

@jakirkham jakirkham mentioned this issue Mar 22, 2018
Closed
@scopatz scopatz mentioned this issue Sep 28, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@msarahan @scopatz @synapticarbors @pelson @ChrisBarker-NOAA @jjhelmus @danielfrg @SylvainCorlay @jakirkham