NVENC VRAM not release after codec switching

[Simba98]

Describe the bug
There are VRAM leakage on NVENC, especially manually switching encoding

To Reproduce

1. Make sure NVENC is enabled. (passed test with https://github.com/keylase/nvidia-patch/blob/master/tools/patch-tester/patch-tester.sh)
2. Running a few days may reproduce it.
3. But, manually switching between H264 and JPEG encoding in client will significantly increase VRAM usage unless there is no more VRAM.

System Information (please complete the following information):

• Server OS: RockyLinux 8
• Client OS: Windows 11
• Xpra Server Version 6.1.3
• Xpra Client Version Tested with 6.1.2/6.1.3

Additional context
There are some warning before VRAM running out.

2024-10-04 13:44:17,923 Warning: image bigger than expected
2024-10-04 13:44:17,924  2880x1536 instead of 2880x1514
2024-10-04 13:44:18,463 Warning: image bigger than expected
2024-10-04 13:44:18,463  2880x1536 instead of 2880x1514

is it related with this issue?
Guessing it is due to 353e5ca

/usr/lib64/python3.11/threading.py:982: UserWarning: module in out-of-thread context could not be cleaned up
  self._target(*self._args, **self._kwargs)
/usr/lib64/python3.11/site-packages/xpra/server/window/video_compress.py:397: UserWarning: module in out-of-thread context could not be cleaned up
  ve.clean()

Something before VRAM running out.

[totaam]

Guessing it is due to 353e5ca

This commit handles decoding of padded frames.
But since you are reporting a memory leak server-side, this has nothing to do with it.

The new increased padding came with a driver update.
The nvenc codec has not changed much in many years - it's possible there is a leak, but perhaps another API change is causing it.

[Simba98]

Guessing it is due to 353e5ca

This commit handles decoding of padded frames. But since you are reporting a memory leak server-side, this has nothing to do with it.

The new increased padding came with a driver update. The nvenc codec has not changed much in many years - it's possible there is a leak, but perhaps another API change is causing it.

I confirmed that I can not reproduce it on Xpra 6.1.2 Server by switching encoding multiple times.