Investigate a method to create or suggest a virtual environment with ipykernel + needed packages for a given notebook

[greazer]

No description provided.

[IanMatthewHuff]

Here is a summary of the customer flow from the PPT slides on getting started env setup.

• Trigger: run code cell without selecting a kernel, select kernel -> create new​
• Ask: Select which Python version you want to use or install​
• Ask: Name your new conda environment (defaulting to “vscode-default”)​
• Create a conda environment from ^ and activate it​
• Look for an environment.yml or requirements.txt in the project​
• Ask: Install packages from (which, if multiple) existing environment definition file?​
• Install packages from ^​
• Select the created env as kernel for the notebook​
• While all this is happening, open “output”

[IanMatthewHuff]

Options (in progress):

• Trigger - When do we suggest this new kernel?
  • User attempts to run code without selecting a kernel
  • User attempts to run code against a global python install
  • User attempts to run code in a new workspace where code has not been run before
  • Instead of running code, if the user has a global python selected for a controller
• Entry Point - What is the UI affordance for creating this new kernel?
  • In the kernel controller selection dialog have a "create new kernel" option
  • When user runs without a kernel selected, instead of popping up kernel selection first offer to create new kernel
  • Lightbulb from unfulfilled imports in an open notebook document
• Python Selection - How to select the python to create the environment with?
  • Quick pick python interpreters (must have one available that we know how to install from, Conda / Pip)
  • Install Python for user
  • Ship micromamba, use that to set up env
• Installing The Kernel - Where to make this new kernel accessible
  • Create a .venv for the kernel relative to the local workspace, install kernel there
  • Install the kernel into the Jupyter global kernel location
• Post Installation - How do we handle the kernel after installation?
  • Nothing special
  • Use as default kernel for the workspace that created it
  • Use as suggested kernel globally when we are not sure

[IanMatthewHuff]

MVP Plan (in progress):

What can we hash out in a few days of work that move us forward here and answer some questions.

Trigger - None
Entry Point - Command Palette command
Python Selection - Pick from Python Extension interpreters
Installing the Kernel - Conda / Pip to create environment. Install ipykernel into env.
Post Installation - Select created kernel

[greazer]

Consider adding a revisitation of whether we install ipykernel without asking to the work here.

One idea that is an alternative to just installing would be to create a cell within the notebook that installs ipykernel for the user. That way they can see what needs to happen and allow it or not. Of course, this would need to be some sort of special case cell. The point is that we have been concerned that users won't like the idea of just having an environment get updated with a package they may not understand the need for and rather than prompting them, is there a more subtle way to ensure the right thing AND right education happens.

[IanMatthewHuff]

Flow for when our command is visible

This scenario is for picking when we should display our env creation command in the kernel picker. Note this is explicitly checking the pip/venv use case. And early second experimentation step will be adding in Conda support as well. Some cases (such as using Micromamba for installs instead of installed python) are out of current experiment scope.

flowchart LR
    start([VS Code Opens])
    endYes([Show the env command in kernel picker])
    endNo([Don't show the command])
    fileOpen{Is a notebook file active?}
    pythonAvailable{Is python available?}
    globalOnly{Are all python controller global?}
    havePip{Is pip/venv available?}
    start-->fileOpen
    fileOpen-- No -->fileOpen
    fileOpen-- Yes -->pythonAvailable
    pythonAvailable-- Yes -->globalOnly
    pythonAvailable-- No -->endNo
    globalOnly-- No -->endNo
    globalOnly-- Yes -->havePip
    havePip-- No -->endNo
    havePip-- Yes -->endYes

Loading

Flow for when our command is executed

flowchart TD
    start([Env creation command is selected])
    multiplePythons{Are there multiple pythons found?}
    pythonPrompt[/Select PYTHON to use. QuickInput /]
    createVenv[Create a .venv in the workspace]
    installIPyKernel[Install ipykernel into venv]
    hasRequirementsTxt{Workspace has requirements.txt file?}
    collectRequirements[Collect PACKAGES from requirements]
    hasImports{Active notebook document exists and has imports?}
    collectImports[Collect PACKAGES from notebook document]
    installPackages[Install PACKAGES if specified]
    installPrompt[/Prompt user for installing PACKAGES with PYTHON. Modal/]
    selectController[Select the created env as active controller]
    start-->multiplePythons
    multiplePythons-- Yes -->pythonPrompt
    pythonPrompt-->hasRequirementsTxt
    multiplePythons-- No -->hasRequirementsTxt
    hasRequirementsTxt-- Yes -->collectRequirements
    hasRequirementsTxt-- No -->hasImports
    collectRequirements-->installPrompt
    hasImports-- Yes -->collectImports
    hasImports-- No -->installPrompt
    collectImports-->installPrompt
    installPrompt-->createVenv
    createVenv-->installIPyKernel
    installIPyKernel-->installPackages
    installPackages-->selectController

Loading

Rough plan

• The command shows up (given pip/venv)
• Prompting for Python version to use (take newest?)
• Create the local .venv and install ipykernel
• Associate the controller with the file
• Add Conda support
• Selecting between Conda / Venv if both are available
• Picking up a local server already running
• 💪 Scan requirements.txt / file for packages to add
• 💪 Install extra packages
• 💪 Detect if we are currently in a conda env and surface that
• 💪 Better progress reporting
• 💪 Define kernels of interest

Open questions

• How should we handle / treat Conda?
  • The association for Conda is a bit different, it's easy to see a .venv as workspace specific, but not the same for Conda. We could check where they are installed or simply have some type of manual step to associate a conda env with a specific workspace
  • I see the second step of this experiment as figuring out the same flow with Conda. Essentially the question that for .venv we answer with "do you only have globals" will need to be generalized for Conda.
• Do we want this controller to be the workspace default?
  • Not sure if the API allows this, but I would want this explicitly created controller to now be the auto suggested / selected controller for new files in the workspace
• How do we scan for requirements.txt? What level? Workspace? Next to file?

[minsa110]

My takeaways per our quick sync today

• The goal for August should be to implement the "rough plan" section as a working prototype (but without focusing on shipping)
• Expand the diagram to capture a broader picture for the experience (e.g. update the flow diagram to include conda, poetry, pyenv, etc.)

A few other open questions

• If this is about an opinionated way of setting users up for success, and our opinion is to have our users NOT use global Python environments, should we even show global Python evns on the kernel picker list? I think that's taking this "opinionated" part to the extreme, but I think it's at least worth a discussion
• How are we defining success? Few suggestions (but we do need some exploration here to ensure we can measure):
  • The desired experience is successful in the top 100 (EDU) notebooks on GitHub
  • Click through rate / feature usage rate over time
  • Using less global Python envs
  • Using more non-global Python envs
  • Successful cell run rate after opening a notebook for new users
• It would be interesting to explore the "pick & choose then confirm" vs. "automate then confirm / edit" experiences. In previous user studies, users had stronger affinity to the UX where they were given options to define the creation of their development environment (i.e. "pick & choose then confirm"), but it would be interesting to learn how well these two experiences do "in the wild"

And another quick thought

• This is kernel setup / creation, but I think we can later explore "managing" it too, e.g. adding package manipulation commands from the integrated terminal back to requirements.txt

[IanMatthewHuff]

Possible flow for Conda => Workspace association. Since we can't just use folder location like with venv instead keep a workspace => env mapping.

If you have previously mapped this workspace to a named conda env, use that. If not, allow the create option. Only before creating, the first option is just to select and existing named Conda env. If that is not the one you want, allow a create option.

[IanMatthewHuff]

Note current flow like so for venv. Note a few things:

1. Notifications are not hooked up correctly, the status bar / progress sections should have more in it
2. A few bits of ipykernel loading and connection were cut, the whole scenario takes a bit longer
3. This was an attempt at "lowest UI" possible. There is already an option to select the python interpreter to create with, and also should possibly be modal confirmation before the env is created.

EditCreateEnv.mp4

[IanMatthewHuff]

A couple more small flow videos with Conda added. Note to not worry too much about UI polish on this still, that side is coming from #11077. Note that progress bars are terrible and double entries in the existing conda list. Also for the video I did some serious shortening of the time sequences (conda env creation takes well > 1 min just by itself, and then 20s or so to connect).

Pick an existing global conda environment:

CondaExistingEnvEdit.mp4

Create a new global conda environment:

CondaCreateEnvEdit.mp4

I consider these more proof of concept for the following:

1. Two different environment creators (venv / Conda) supported
2. Ability for both creator to control when they are available (venv = python with venv / pip, conda = conda on the command line)
3. Ability for both creators to indicate if they already have a locally associated kernel (venv = local in workspace, conda = env previously selected)
4. Ability for both creators to create a new locally associated kernel (venv = install locally, conda = associate with existing or install globally and associate)
5. Creators can both install ipykernel (could be expanded to installing from requirements.txt or file imports)

[IanMatthewHuff]

Closing as python extension provides this.