[FEA] Ability to limit total GPU memory used

[jlowe]

Is your feature request related to a problem? Please describe.
Sometimes the plugin needs to share the GPU with another process, and the plugin's current behavior of aggressively allocating most of the GPU memory for the RMM pool is hostile to that setup. Even if one manually configures the initial RMM pool size to be lower, it will grow without bound if the plugin requests that much memory. It would be nice to be able to put a hard limit on the RMM memory size.

Describe the solution you'd like
A plugin config that can be used to set a limit on the total size of the RMM memory pool.

Describe alternatives you've considered
Being able to share the same RMM pool between the processes would be ideal, then we wouldn't have to hard-partition the GPU like this. However even if that feature existed, we would still need some guarantee the plugin wouldn't end up consuming all of the shared RMM pool just as the other process needs it.