Ark: Decouple LSP and kernel?

[lionel-]

Currently the LSP and the Jupyter kernel are quite tangled with each other through the use of r_task().

• This has performance and safety implications, see Ark: Infrastructure: Reduce concurrent R evaluations to a minimum #2322

• This prevents LSP features from working while the R session is disconnected or shut down. A shutdown R can happen early at startup during runtime discovery, or if the user disabled automatic startup, or if they closed the session. I've noticed that it's puzzling to have different indentation behaviour depending on the availability of the LSP for instance. This sort of issue will become more visible as we ramp up our formatting capabilities through the LSP.

If we make sufficient progress with #2322 (avoid concurrent R calls) and #2321 (use static analysis where possible), we should be able to pull out the LSP from the kernel and maintain a permanent session in the frontend. The LSP would then communicate with the kernel through a Jupyter comm, e.g. to receive updates about global console scopes (global env, search path, etc) between top-level R evaluations.

One problematic aspect is the generation of help pages which does require calling R. We'd need a side session to populate these. It's unclear what we could do regarding help from devtools during development though.

A middle ground would be to decouple only parts of the LSP and run two instances, one in the frontend and the other in the backend. This would likely require implementing our own VS Code providers to select the correct instance.