pip dependencies in conda recipe

[arsenovic]

is there a way to include dependencies in a meta.yaml that are not yet available as conda packages?

or is the solution to go and build conda packages for all deps?

[jakirkham]

What sort of dependencies are you trying to include?

Python? AFAIK, no. I know there use to be an issue for listing PyPI dependencies in a package and for the life of me I can't find it. If it is Python, you can try the following:

• conda pipbuild <pypi_package> - Builds packages for the package listed and any dependencies it can't find. When it works, it is amazing, but there are edge cases it will fail on.
• conda skeleton pypi <pypi_package> - Gives a rough start at a Python package. There are variants for some other non-Python languages. Not recursive like conda pipbuild, but a bit more stable.
• Hunt down a recipe. There are tons here ( https://github.com/conda/conda-recipes ). Not to mention, most people put their recipes in a repo on GitHub and put the least restrictive licenses on them.
• Finally, write your own. Given the numerous examples and methods to get an easy start. It is pretty easy to tweak something to get what you want.

[connie]

Is it possible to use --pypi-url to point to a pypi.anaconda.org/username channel because there is a package there I want to use, rather than from pypi.python.org? I can't seem to figure out how to get it to work.

[kynan]

Supporting pip dependencies would be great particularly for conda forge. If you want to add a new recipe you currently need to make sure all dependencies are available via conda.

[znicholls]

Was there ever any progress on this? In order to have a pip dependency, at the moment it seems like you need to go through your entire dependency tree and make sure every package is available through conda. This seems like a lot of mucking around for what intuitively feels like a simple operation (maybe that's naive).

[jonas-eschle]

Hi all, is there any news? Or anything contrary? It is a huge problem if not all packages have an anaconda build, e.g. dependencies: it means we can't put our package on conda neither. I know it ain't the ideal way, but the realistic; otherwise we're constantly deprecating conda to the users as a dependency (sometimes a single one) is missing or outdated.

[analog-cbarber]

The problem is that conda package dependencies only refer to other conda packages. Conda and pip are two separate packaging systems and there is no guarantee that a pip version of a package is even compatible with the conda version. Although uncommon, there are plenty of examples where installing a pip package into a conda environment can result in a corrupted environment. So if you want to publish a conda package, you really need to make sure that all of its dependencies are available from some conda channel. If necessary, that means repackaging pip packages and publishing them in your own channel.

If you just want people to be able to create a conda environment with pip dependencies, then you can list them in an environment file. For instance see https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-file-manually.

[github-actions]

Hi there, thank you for your contribution to conda-build!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue in the latest version of conda-build

2. Comment that the issue is still reproducible and include:

• What version of conda-build you reproduced the issue on
• What OS and version you reproduced the issue on
• What steps you followed to reproduce the issue

3. It would also be helpful to have the output of the following commands available:

• conda info
• conda config --show-sources
• conda list --show-channel-urls

NOTE: If this issue was closed prematurely, please leave a comment and we will gladly reopen the issue.

In case this issue was originally about a project that is covered by the Anaconda issue tracker (e.g. Anaconda, Miniconda, packages built by Anaconda, Inc. like Anaconda Navigator etc), please reopen the issue there again.

Thanks!

[jonas-eschle]

Conda and pip are two separate packaging systems and there is no guarantee that a pip version of a package is even compatible with the conda version.

Yes, but shouldn't this be the task of the installing package? In this case pip?

If you just want people to be able to create a conda environment with pip dependencies, then you can list them in an environment file.

Not quite, because it also means that I cannot distribute my package with conda either, since conda would require me to have all the dependencies in conda.

I do understand the reluctance of making this possible, but I also see the negative effects of it: packages not getting published on conda instead. And that's not better either.

[analog-cbarber]

The problem is that a conda package simply has no way to declare a dependency on a pip package, so there is no way for conda-build to include them unless perhaps you want conda-build to magically generate a post-install script.

But even if you could do that, it isn't even guaranteed to work. There are plenty of cases in which installing a pip package will result in corrupting your conda environment. Pip will happily overwrite files installed by conda with versions that are not binary compatible. Pip doesn't know or care anything about conda dependencies.

So yes, you DO need to make sure all your conda dependencies are in conda if you want conda install to work. That means that sometimes you have to build extra conda versions of those pip packages and publish them on a channel you control. It usually isn't all that difficult, but most people don't bother.

[jonas-eschle]

I agree on the technical difficulties, especially with pip and stuff and the proposed solution sounds nice on paper.

...and publish them on a channel you control. It usually isn't all that difficult, but most people don't bother.

This doesn't work because we cannot add a channel in a recipe (#532).
If that would work, I could implement this solution, but the whole solution currently reads that you package it, publish it on a channel you want *and then force the user to add this channel when installing. A user cannot simply install package A from the usual (say, conda-forge) channel but also needs to add channel Beta because of dependency B.
Now, it doesn't need to stop there! The install will fail because B depends on C that is available in Charlie, so as a user, you also need to add this channel and so on. Now, is a user expected to chase down all of this dependencies until he gets all the channels right? This doesn't seem to be how a package manager should work, does it?

That's the problem: it's not as easy as publishing it in a channel. The user will need to chase down every single dependency in different channels.

I don't advocate for the pip solution, but for either the pip solution or getting #532 in. Because currently, if the package is not in a standard channel and the author doesn't add it, it is not possible to add it to a channel, as far as I understand the proposal.

Or what would you suggest to circumvent this problem?

[analog-cbarber]

Since conda does not support pip package dependencies (and doing so would be questionable given the potential for pip to corrupt your conda environment), there is nothing conda build can do to make conda install work.

What channels you may use to publish packages is an entirely separate issue.

But in an enterprise setting, you typically will have private channels that you use to publish and install packages and users will be used to specifying those channels. You may have wrapper scripts to make this a little easier or you might have everyone configure those channels as defaults. In that case, it is not such a big deal to publish additional packages in channels you control.

Now if you wanted to publish a package on conda-forge that depends on pip packages it is a different story. In that case, you probably do not want to introduce any other channel dependencies. Ideally, you should work with the authors of the pip package in question to get that package published on conda-forge. Most projects will welcome your help in doing that because it will benefit everyone and fix the issue. If you don't have the time or interest in doing that, you have a number of other choices of various levels of hackiness:

• If the package is only used in some situations, you can make installation optional, and modify your python code to dynamically import the package in question. Then tell your users that the relevant features will only work if they have installed the relevant package.

• Include a post-install script in your recipe that will do the pip install.

• Just pip install the package as part of your build so that it is included in your own package. This is only possible if the licenses allows for that and is compatible with your license.

[mfansler]

• Include a post-install script in your recipe that will do the pip install.
• Just pip install the package as part of your build so that it is included in your own package. This is only possible if the licenses allows for that and is compatible with your license.

Please don't do this. If the dependency package is so easy to install with pip, then get a build going on Conda Forge. And if you need help, then post a package request in the conda-forge/staged-recipes/issues.

The main problem with making a multi-module super-package or side-loading in a post-install script is that this subpackage will not be tracked as a real dependency and when it does eventually get added to the Conda ecosystem, you'll end up with package clobbering.

[analog-cbarber]

Well, you are probably right that most users don't care about how the sausage is made and will be happy as long as things get installed and appear to work. But 99.99% of users also will be really unhappy when the code behaves incorrectly or crashes and will be even more unhappy that there will be no indication what the reason for the failures is.

I suppose all of this would be much simpler if conda just ignored pip entirely and didn't try to support it at all....

But if you think there is an easy solution to this problem then implement it and submit a PR....

[analog-cbarber]

The thing is you specifically said there was a simple solution and are mad this issue was closed, but now you admit that there really isn't. So what are you really complaining about?

The fact is that for this particular issue there IS no magic fix regardless what you think of conda's code complexity because it really has to do with the fact that pip and conda are totally different packaging systems with different package repositories with no guarantee that packages will be compatible between them. You know perfectly well that pip will happily blow away conda-installed files without any warning whatsoever, and you are upset that conda does not magically fix that problem?

The real solution to this problem is to avoid using both conda and pip to install your packages into the same environment, it definitely is NOT to make the problem worse by allowing conda packages to actually have runtime dependencies on pip packages.

Note that no one is complaining that conda does not allow you to express dependencies on any number of other packaging systems such as npm, maven, and the like.

[jonas-eschle]

(remark: I am in no way associated with conda)

The code installed by conda mostly behaves incorrectly. It's such a garbage package manager I've just thrown my hands in the air trying to work with it. It's hopelessly bad, beyond repair or redemption.

@wvxvw these comments are completely out of line! Please behave and have the necessary respect in an open-source community where many people dedicate their freetime to comment and resolve issues.
If you don't like it, don't use it. I personally really like conda (mamba) and the concept of it, it has brought huge simplifications to us to install packages that aren't purely Python. Yes, it's not perfect. But the problem here is clearly your attitude, to expect perfect software for exactly your use-case, not conda itself.
Please reconsider your words, it seems that you have a lot of irrational anger. Don't blame someone who tries to solve your problems (but maybe doesn't fully manage) for your actual problems. No one forced you to use conda.

@analog-cbarber just for some technical discussion

The real solution to this problem is to avoid using both conda and pip to install your packages into the same environment

This is just a bit tricky. I absolutely see the issues and I agree. Nothing wrong with the argument that users will be upset if it doesn't work, I fully get that. But... it implies that if a single package is not available in conda, I should install everything in pip, right? And that's just not so satisfying: what if a library owner doesn't create a conda package? No chance. Because I cannot simply create it and add it (#532), which is what you can do in pip (because you can upload a package and specify it as a dependency. That does not work with conda.

In this sense its not that conda and pip have different mechanics. It's that conda is more restrictive in how you can get packages in than pip.

To be specific, a simple example: A depends on B. But B is not in conda nor pip. For pip I can simply create a CD by cloning the repo (assuming the license allows it) and make my own pip package B2 and add this into the requirements. While I could create my own conda package, I cannot put it into the requirements because I can't upload it to conda and I can't add my channel.

Again, there are reasons for that. But undeniably, conda is more restrictive than pip. And, as you said, be exclusive means "don't use conda" (except if all packages are there). People would love to use conda more, basically having the less restrictive nature of pip but conda-style packages. This are perpendicular issues: one is policy, the other is software.

And it's the former, the strict policy, that people are actually unhappy with, trying to find solutions, and the answer they get is "you can't use conda for that package" and "avoid using both conda and pip" -> use pip only.

Note that no one is complaining that conda does not allow you to express dependencies on any number of other packaging systems such as npm, maven, and the like.

This is true. Because pip is the de-facto Python standard package installer, it's where all packages are. Every other package manager provides merely a subset of the packages. The packages missing in conda are in pip, but not in (usually) apt, maven...

I'd leave it like this and want to emphasize again how great conda is! It's a game changer on so many levels, thank you all very much for this.
And exactly therefore it is bitter that the stronger security policy - which has nothing to do with bugs or software - is preventing the use of it if a single package cannot be made available on conda.

[analog-cbarber]

@wvxvw

If you think there is even an easy "partial" solution, then go ahead and implement it in a fork.

I don't think that this problem is inherently solvable, but if you want an easy "partial" solution that already works,
just give your package a post link script to install any pip dependencies.

In any case, I don't think that attacking the project and its maintainers is likely to get you the results that you want.

[jonas-eschle]

You are not the opensource community. Don't hide behind the back of people you have nothing to do with.

I have very little respect for you individually. This has nothing to do with millions of people, most of whom I don't know, but, generally, appreciate the category they belong to.

I can just repeat myself here:

...[have respect] in an open-source community where many people dedicate their freetime [...]

You are here in an open-source community. Please behave. There is no need to attack individuals, there is no need to make wild accusations that nobody ever claimed.

Let me widen that: don't behave like that in any working environment. Be constructive and objective, talk about issues, not about people.

This is another lie that you and people like you repeat when, instead of being helpful, you just want to dismiss the criticism. I use conda not because I like it. Similar to how I use English not because I like it in particular.

There is criticism and there is insult. I read only the former. And about "people lying". Talk about technical problems in technical language, about matters, not people. Then you're very welcome to bring up anything you encounter

Programs are tools of communication, and I cannot unilaterally select a communication tool or a protocol -- not only do I need at least one more person to talk to, in the case of conda, I need to talk to hundreds of people.

Had you had a little bit of understanding of the reason for open source software to exist, you might have known this already. Unfortunately, you are just riding a wave of popularity which randomly made you the gatekeeper of the communication tool.

They are. Conda is a tool that helps a lot in installing and managing installed packages. It's not - that maybe comes as a surprise - the "I solve all the installation problems you ever had". No. There is no such tool. And maybe you're even bound from your employer or similar to this tool. Be mad at them if it's the wrong tool for the job. Don't hate cars because someone tells you to fly with them. The wrong tool for the wrong job doesn't make the tool bad.

That said, please adjust moving forward and, again, talk about technical issues in objective, technical language, not about people, don't insult.

[tnabtaf]

Hi All,

Several recent posts in this thread were reported to the conda Code of Conduct (CoC) Committee. After establishing contact with the posts' author a decision was made to block that person for an extended period of time in all conda spaces. See the full public report for details.

---

We encourage everyone to keep the conda ecosystem a welcoming and harassment-free environment. If you notice or experience a violation of our Code of Conduct, we encourage you to gently remind offenders of our rules (as many did here, this often works to deescalate), or to report the incident to the CoC Committee for review.

[kynan]

Thank you, @tnabtaf. Can I suggest also adding a CODE_OF_CONDUCT.md to the conda-build repository itself and referencing it from the README? Further it would be great to reference the Code of Conduct on docs.conda.io

GitHub's automatic reference to the Code of Conduct below the comment authoring box is unfortunately tiny and easy to miss.

[tnabtaf]

Thanks @kynan, that is a great idea. Sometime in the next month or two we will move the Code of Conduct out of the conda-incubator GitHub organization and into the conda organization. That will be an excellent time to update all repositories as well.

Thanks for the suggestion.