Support poetry virtual environments

Motivation

Support for poetry environments is currently our highest upvoted feature request on github. It attempts to solve problems with all predating tools to manage Python packages: virtualenv, pip, pipenv, etc by being an all-in-one tool, and hence is becoming increasingly popular.

Scope

• Discover environments created using Poetry
• Identify if a provided custom interpreter path belongs to a Poetry environment
• Install dev tools using Poetry when related poetry environment is selected

Things we decided we need not address,

• Activate environment using poetry CLI: poetry shell command is similar to pipenv shell, which has its issues #4404. Besides, we don't need to activate these environments via poetry, as it manages venv/virtualenv environments which can directly be activated using the activate scripts.
• Run scripts using poetry CLI: poetry run command can be useful to run scripts if for some reason activating poetry environment doesn't work properly. But that is not the case observed with these environments for now, so we need not use it.

Design

Identifier

Global poetry environments

Poetry environments doesn't have a .project file equivalent like Pipenv, using which we can locate the project for which the current environment is created; hence we can't say for sure whether it's poetry environment by just looking at the environment path.

• Our best guess is to use the environment name pattern which Global poetry environments follow, as evident from the code here:

<sanitized_env_name>-<cwd_hash>-py<major>.<micro>

Implementation details behind <sanitized_env_name> and <hash of cwd> are too much rely upon, so for our purposes the best we can do is verify if the pattern matches:

<anything>-<anything>-py<number>.<number>

• Should be a virtualenv/venv type environment.

If this doesn't work well, we can follow other alternatives mentioned in the spike.

Local poetry environments

• Environment folder is named .venv
• One of the parent directories contain pyproject.toml which contains a poetry section.
  • We can directly run poetry check to verify this assumption.
  • If running command for every .venv-named folder turns out to be expensive (probably won't be), look for pyproject.toml file in the current workspace folder. (the correct thing would be to look up all the parent directories as well, but that is expensive). If found, read the file and see if contains the string [tool.poetry].

Discovery

Location for poetry environments is decided by the virtualenvs.path setting. It defaults to {cache-dir}/virtualenvs where {cache-dir} is another setting with its own defaults. Poetry looks at the following entries in order to get a setting,

• Environment variable: Every setting key has an associated environment variable which begins with POETRY_ (more). For eg. for the virtualenvs.path setting the environment variable to look into is POETRY_VIRTUALENVS_PATH.
• Local config: Stored in poetry.toml which is located besides pyproject.toml.
• Global config: Stored in these locations, in a file named config.toml. Parse file using npm package toml to get the setting value.

Unfortunately poetry CLI doesn't provide APIs individually query for these 3 types of configs.

We have two locators to search for all these locations for,

• Global poetry locator:

  • Env variable POETRY_VIRTUALENVS_PATH
  • Global virtualenvs.path setting

  Now we look at {cache-dir}/virtualenvs, possible {cache-dir} values are,

  • Env variable POETRY_CACHE_DIR
  • Global cache-dir setting
  • Default setting value based on OS

• Workspace poetry locator

  • Local virtualenvs.path setting
  • Local cache-dir setting
  • Looks into .venv folder if it exists

Use identifier, pyvenv.cfg files and other utils to get type, version and other details. Each locator extends FSWatchingLocator for file watching, and uses the existing ILocator interface.

Installer

It uses the existing IModuleInstaller interface which exposes two methods:

• isSupported: Returns whether we can install dev tools using poetry for a particular resource. If the selected environment for the resource is the active poetry environment which was created for the resource, return true, otherwise return false. Steps,

  • Get the associated poetry environment for the resource using poetry env info -p command.
  • Verify if it's the same as the selected environment for the resource.

  Running command here is not expensive as we already have the extension activated, and this only happens after user-consent. But we can use other alternatives if for some reason this doesn't work out. Users can specify the poetry executable to use using the python.poetryPath setting.

• installModule: Use poetry add --dev <moduleName> to install dev tools. Add --allow-prereleases for packages for which released version is not available yet. (Example, black)

APIs

If this new work exposes public APIs for consumption by others, document them here. Note that the APIs may be emergent (TDD can help with this), so this section may be updated later, unless delaying the final design of the APIs would be blocking clients of the APIs, in which case we may need to front-load the work in this section. Sometimes APIs are documented separately in reference documentation, but it makes sense to start with the design doc while the API is still being agreed on.

Special Considerations

Discussion of novel algorithms used or designed, cryptography usage, threat model, privacy (including use of PII), scalability, resource requirements, etc.

Telemetry and Experimentation

How will we measure success of the feature? Are there experiments we want to conduct that we need to account for during implementation? What telemetry might we need? Are we going to want dashboards for product health and feature usage, and will we have the data for those?

Testing Strategy

How will the feature be tested? Where are the right “seams” for unit testing? Are there integration test concerns? Will manual testing be required, and who will carry that out at what cadence? How will coverage data be collected and processed?