PEP684 Subinterpreter support

[sklam]

Feature request

PEP684 will allow subinterpreters to be properly isolated (separate module state, separate GIL) in Python 3.11. However, extension modules will need to be modified properly support subinterpreters; e.g. C globals needs to be protected from multiple interpreters.

Numba needs to decide on how multi-interpreters are supported.

Options:

0. Unsupported.
1. Complete isolation. Every interpreter is treated as if they are separate process. This will require extension modules to be isolated and LLVM JIT engine to be created per interpreter.
2. Shared LLVM JIT state. Avoid JITing the same function multiple times. Share JIT engine in the process.

Other things to consider:

• Managing JITed function lifetime. Interpreters can be created dynamically and destroyed throughout the process lifetime.
• GPU resources?
• Rethink all atexit use

Additional reading:

Related discussion for NumPy: https://mail.python.org/archives/list/numpy-discussion@python.org/thread/A3CWGUGK23D33ECXCX6C7JN6OK273KAJ/
How to isolate extensions: https://peps.python.org/pep-0630/