Add support for nb_conda_kernels

[jeiche]

If you're using nb_conda_kernels, voila fails to execute the notebook since it fails to find the kernelspec due to the fact that it implicitly assumes you're using Jupyter's KernelSpecManager. I propose making the following code snippet change in voila/app.py to address the issue.

Old:

from jupyter_client.kernelspec import KernelSpecManager

New:

try:
    from nb_conda_kernels import CondaKernelSpecManager as KernelSpecManager
except ImportError:
    from jupyter_client.kernelspec import KernelSpecManager

[SylvainCorlay]

Hi @jeiche this is really interesting.

In fact, the main issue is that nb_conda_kernels is an extension to the notebook server and was not updated to work with jupyter_server that voilà is using (and soon jupyterlab and the like).

For more context, the issue with that extension is that it completely overrides the kernel manager. The new Jupyter Server includes a pluggable kernel discovery mechanism and I think that using it to make it work with conda environment is a really good fit.

cc @kevin-bates.

[danlester]

I've also seen this problem from Voila users, and because nb_conda_kernels needs updating really, I've recommended that users 'feeze' their kernels by removing nb_conda_kernels and just registering existing kernels manually. Of course this means that new kernels won't be discovered in Jupyter.

There is a bit more info here for anyone trying this 'workaround':

https://cdsdashboards.readthedocs.io/en/stable/chapters/troubleshooting.html#conda-kernels-not-found-by-voila

[kevin-bates]

The new Jupyter Server includes a pluggable kernel discovery mechanism and I think that using it to make it work with conda environment is a really good fit.

@SylvainCorlay - Just to clarify and manage expectations, Kernel Providers have not been merged into jupyter_server and likely won't be merged until 3.0 and the JEP has been ratified, etc. That said, I agree Conda kernels make for a perfect example of a kernel provider and work had even been started on an early prototype (just too early, unfortunately).

Taking a brief look at the code, it seems like Voila is going to have to make changes to adopt Kernel Providers if/when they're available since they'll replace things like self.kernel_spec_manager (assuming that's being derived from the underlying notebook server). Is that on the radar?

[mcg1969]

Version 2.3.0 of nb_conda_kernels offers a mechanism for supporting voila thanks to a change offered by @fcollonval; see anaconda/nb_conda_kernels#172

@kevin-bates is correct that proper support is going to have to wait for true kernel provider support to be incorporated into Jupyter itself.

[mcg1969]

I personally am partial to my monkeypatching approach
anaconda/nb_conda_kernels#104
But @fcollonval prevailed upon me to take his more prudent approach first.

[SylvainCorlay]

@SylvainCorlay - Just to clarify and manage expectations, Kernel Providers have not been merged into jupyter_server and likely won't be merged until 3.0 and the JEP has been ratified, etc. That said, I agree Conda kernels make for a perfect example of a kernel provider and work had even been started on an early prototype (just too early, unfortunately).

Yes, that was just me looking a bit too far forward :) thanks for chiming in.

I personally am partial to my monkeypatching approach anaconda/nb_conda_kernels#104
But @fcollonval prevailed upon me to take his more prudent approach first.

Thanks for working on this. I really hope that the kernel provider mechanism will allow us to not have to monkey-patch!

In the mean time, the solution on our side should be to document the behavior and point to the section in the documentation.

[kevin-bates]

I really hope that the kernel provider mechanism will allow us to not have to monkey-patch!

Although Kernel Providers may have their own set of pain points, discovery and launch functionality is isolated and contained within each provider and each can co-exist with other providers.

One area that provider authors should be aware of though, is the use of kernel.json to house a kernelspec's config settings is "owned" by the default kernel provider. Attempts by other providers to reuse that name will result in double discovery of the kernelspec - one by the default, the other by the provider. One thought I had was adding a provider_id to kernel.json files for non-default providers, but this would require the file be opened on each discovery cycle - which is fairly frequent. This could be mitigated with some "hashing and caching" however. As of now, custom providers need to use a different name. (The remote providers I've built as POCs use <provider_id>_kernel.json as their spec file since they're also based on today's model.)

[fcollonval]

A tutorial to demonstrate how to set up nb_conda_kernels has been published.