Adding support for activating conda environments in the kernel.json launch script?

[ericdill]

Hi ipykernel team! When using a Jupyter notebook with a conda environment, !conda commands do not operate on the conda environment that the user's kernel is running in. At least, not without some surgery to the kernel.json file.

Below, I motivate the use-case for why you want to properly activate a conda environment before launching a kernel out of it. Is there interest from the ipykernel team in accepting a pull-request that adds this functionality to the install subcommand on ipykernel?

First pass at a proposed API is to add a --conda flag to the call so that it would look like python -m ipykernel install --user --name <kernel name> --conda. But, before any work is done to add this to the ipykernel code base, let's discuss in an issue.

Motivating the problem

@parente and I found that it was a pretty common user expectation that they could add a package to their environment when running in a jupyter notebook by executing !conda install <whatever package>. As such, we (actually mostly @parente) ended up writing kernda so we could add source activate <conda env> to the launch script in the kernel.json file in a reliable way.

The usage is as follows:

1. create your conda environment: conda create -n moviepy moviepy ipykernel -c conda-forge
2. activate it and create your kernel: conda activate moviepy, python -m ipykernel install --user --name moviepy

At this point, let's inspect the kernel.json file:

{
 "argv": [
  "/home/ericdill/miniconda/envs/moviepy/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "moviepy",
 "language": "python"
}

This kernel will successfully launch a kernel out of the moviepy conda environment, but it notably is not activating the environment. This has two main ramifications:

1. There are scripts in ./activate.d/ and ./deactivate.d/ that, most commonly, set and unset environmental variables that can be required for proper conda behavior. I'll track down a few of these packages to provide some concrete examples
   a. https://github.com/conda-forge/proj.4-feedstock/blob/master/recipe/scripts/activate.sh
   b. https://github.com/conda-forge/qgis-feedstock/blob/master/recipe/scripts/activate.sh
   c. (will continue looking for more)
2. There can be a user expectation that !conda operates on the conda environment that the kernel is running out of.

So, let's use kernda to edit this kernel.json file to add source activate to the script:

conda install kernda -c conda-forge
kernda --display-name 'moviepy [py37]' /home/ericdill/.local/share/jupyter/kernels/moviepy/kernel.json

This will output the following that you can either redirect into the proper file (or you can use the -i flag to overwrite the original):

{
  "argv": [
    "bash",
    "-c",
    "source \"/home/ericdill/miniconda/bin/activate\" \"/home/ericdill/miniconda/envs/moviepy\" && exec /home/ericdill/miniconda/envs/moviepy/bin/python -m ipykernel_launcher -f '{connection_file}' "
  ],
  "display_name": "moviepy [py37]",
  "language": "python"
}

At this point we will have a kernel that has proper conda activation and gives the user the !conda functionality in their notebook.

edit: add activate.d example

[blink1073]

I like this idea overall. I thought the new pattern was conda activate cross-platform?

[ericdill]

conda activate is, at least for now, best used interactively because it is highly dependent upon the configuration of your shell (powershell, included). @isuruf pointed out on the conda-forge gitter channel that there's also a conda run feature that would make this implementation very clean, but I'm not sure what the status on that is. Last I heard it was still a beta feature. I'll check with the conda team and see if they think that it's ready for "production" use in this context. For now, I think the following implementation options are on the table:

1. Ideally we can use conda run which would make this invocation platform agnostic, so the argv bit would look like this:

"argv": [
  "bash", "-c", "conda run -n <name of env> python -m ipykernel_launcher -f '{connection_file}'"
],

2. Otherwise, we'd need to fall back to the platform-specific source /path/to/activate <env name> on unix and /path/to/activate <env name> on Windows (or whatever the syntax is 🙂 )

Let me get a response from the conda folks and I'll update this issue

[blink1073]

Yeah, on Windows if using cmd or Powershell is /path/to/activate but some of us only use Git Bash on Windows. 😀

[flutefreak7]

I was just having what I think is a related issue that I documented on stack overflow...

https://stackoverflow.com/questions/57210940/access-conda-python-kernel-from-non-conda-jupyter-installation

Basically I have my main python installation from python.org using pip to install jupyter. I then need to use a conda environment for something and I would like to be able to use my conda installation from my existing Jupyter install as a kernel. I successfully added the kernel to the list and can select it from within JupyterLab. Unfortunately, upon kernel activation, zmq fails to load with DLL errors because conda's library\bin folder isn't on the path.

If I instead first run the anaconda prompt which activates the base environment of my miniconda installation, then explicitly run my python.org/pip -based Jupyter server, I can now run the conda-based kernel or the python.org based kernel just fine. I think it's because activating the environment enables the conda DLL's to be found.

I understand that nb_conda_kernel is generally cited as the way to most easily make conda environments available, but nb_conda_kernel isn't available to a python.org / pip installation of jupyter.

[dkliu1]

I have the same problem trying to setup an environment/kernel (gis) to be launched from jupyter lab installed on the default environment (base).
The following works when running on Anaconda Powershell

conda activate gis
python
import pyproj

However, when I launch from jupyter lab the gis python kernel installed via

python -m ipykernel install --name gis --display-name "Python 3 (gis)"

I get the following error

import pyproj
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-98cb605ea9de> in <module>
----> 1 import pyproj

C:\ProgramData\Anaconda3\envs\gis\lib\site-packages\pyproj\__init__.py in <module>
     68 import warnings
     69 
---> 70 from pyproj import _datadir
     71 from pyproj._list import (  # noqa: F401
     72     get_angular_units_map,

ImportError: DLL load failed: The specified module could not be found.

[TormodLandet]

I have the same problem with shapely / geos on Windows. If the conda environment is not loaded, many packages do work by just specifying the full path to python in "kernel.json", but importing "geoviews" fails since some of the shapely / geos DLLs on windows do not work correctly without loading the conda environment first.

My simple work-around is to launch the kernel via a bat file that loads conda

Create a launch.bat file in the kernel directory containing the following (geoviews is the name of my environment):

call C:\anaconda3\condabin\conda.bat activate geoviews
python -m ipykernel_launcher -f %*
call C:\anaconda3\condabin\conda.bat deactivate

Then specify a kernel.json file as follows (notebook is the environment running jupyter-lab in this case):

{
 "argv": [
   "cmd.exe",
   "/C",
   "C:\\anaconda3\\envs\\notebook\\share\\jupyter\\kernels\\geoviews\\launch.bat",
   "{connection_file}"
 ],
 "display_name": "geoviews",
 "language": "python"
}

I tried to do this without the bat file, but there seems to be a difference between launching conda.bat with and without "call" in front. By using cmd.exe and "call" it works.

PS: I do not have conda installed in the PATH (just as recommended by the installer) so this should work for must people, I think ...

[dkliu1]

Thanks TormodlLandet! I tried your method and it works.

[rahulpshah]

Any progress on this issue? Would love to contribute or have support for this feature

[mathause]

nb_conda_kernels seems to also activate the conda environment - however, only for auto-detected kernels and not for installed kernels... The kernel.json would look like

{
 "argv": [
  "python",
   "-m",
   "nb_conda_kernels.runner",
   "/usr/local/Miniconda3",
   "/home/mathause/.conda/envs/test_basemap",
   "/home/mathause/.conda/envs/test_basemap/bin/python",
   "-m",
   "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python (test_basemap)",
 "language": "python"
}

[beckermr]

bump here! If someone has a clear path, i'd be down to code a bit.