A tool for enriching the output of nvidia-smi.
Install
pip install nvidia-smi2
Run
nvidia-smi2 [-l [length]] [-u [username]]
print GPU utilization with usernames and CPU stats for each GPU-utilizing process
-l|--command-length [length] Print longer part of the commandline. If `length'
is provided, use it as the commandline length,
otherwise print first 100 characters.
-c|--color Colorize the output (green - free GPU, yellow -
moderately used GPU, red - fully used GPU)
-u|--user Name of user to summarize
Or run from src
pip install termcolor
chmod a+x nvidia-htop.py
./nvidia-htop.py [-l [length]] [-u [username]]
print GPU utilization with usernames and CPU stats for each GPU-utilizing process
-l|--command-length [length] Print longer part of the commandline. If `length'
is provided, use it as the commandline length,
otherwise print first 100 characters.
-c|--color Colorize the output (green - free GPU, yellow -
moderately used GPU, red - fully used GPU)
-u|--user Name of user to summarize
Note: for backward compatibility, nvidia-smi | ./nvidia-htop.py [-l [length]] [-u [username]] is also supported.
Note: running inside a container (docker, singularity, ...), nvidia-smi can only see processes running in the container.
rnd@rnd:~$ nvidia-smi2 -l
Wed Jul 12 10:41:16 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A |
| 55% 73C P2 88W / 280W | 3248MiB / 11176MiB | 64% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce ... On | 00000000:02:00.0 Off | N/A |
| 59% 67C P2 91W / 280W | 9397MiB / 11178MiB | 7% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| GPU PID USER GPU MEM %CPU %MEM TIME COMMAND |
| 0 1242 root 9MiB 0.0 0.0 175 days /usr/lib/xorg/Xorg vt1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| 0 1966 gdm 4MiB 0.0 0.0 175 days /usr/bin/gnome-shell |
| 0 6963 anonym1 2365MiB 2.9 3.1 7 days python train.py |
| 0 8200 root 135MiB 0.1 0.4 117 days python3 app.py xxxxxxxxx |
| 0 12474 rnd 135MiB 0.1 0.8 11 days python3 main.py xxxxxxxxxxx |
| 0 19369 root 455MiB 62.4 2.5 20 days python3 read_rtsp.py --path_stream xxxxxxxxxxxxxxxxxx |
| 0 30502 root 135MiB 0.0 1.8 16:24:10 python build_index.py |
| 1 1242 root 4MiB 0.0 0.0 175 days /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth xxxxxxx |
| 1 19422 root 455MiB 86.7 3.7 20 days python3 read_rtsp.py --path_stream xxxxxxxxxxxxxxxxxxxxxxx |
| 1 25255 anonym2 6917MiB 128 4.3 22:30:29 python train_image_classifier.py xxxx |
| 1 27877 root 2017MiB 9.4 1.4 15 days python consumer.py |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| USER TOTAL GPU MEM TOTAL %CPU TOTAL %MEM |
+----------------------------------------------------------------+
| root 13MiB 0.0 0.0 |
| gdm 4MiB 0.0 0.1 |
| anony1 1086MiB 0.5 3.2 |
| anony2 6930MiB 478.8 9.5 |
| anony3 7418MiB 111.0 5.4 |
|----------------------------------------------------------------|
| all-users 15451MiB / 22528MiB 590.3 18.2 |
+----------------------------------------------------------------+
rnd@rnd:~$ nvidia-smi2 -l -u root
Wed Jul 12 10:51:06 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A |
| 58% 69C P2 106W / 280W | 3248MiB / 11176MiB | 5% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce ... On | 00000000:02:00.0 Off | N/A |
| 59% 65C P2 80W / 280W | 9397MiB / 11178MiB | 7% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| GPU PID USER GPU MEM %CPU %MEM TIME COMMAND |
| 0 1242 root 9MiB 0.0 0.0 175 days /usr/lib/xorg/Xorg vt1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| 0 8200 root 135MiB 0.1 0.4 117 days python3 app.py xxxxxxxxx |
| 0 19369 root 455MiB 62.4 2.5 20 days python3 read_rtsp.py --path_stream xxxxxxxxxxxxxxxxxx |
| 0 30502 root 135MiB 0.0 1.8 16:33:59 python build_index.py |
| 1 1242 root 4MiB 0.0 0.0 175 days /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth xxxxxxx |
| 1 19422 root 455MiB 86.7 3.7 20 days python3 read_rtsp.py --path_stream xxxxxxxxxxxxxxxxxxxxxxx |
| 1 27877 root 2017MiB 9.4 2.2 15 days python consumer.py |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| USER TOTAL GPU MEM TOTAL %CPU TOTAL %MEM |
+----------------------------------------------------------------+
| root 3210MiB 158.6 10.6 |
|----------------------------------------------------------------|
| all-users 15451MiB / 22528MiB 590.3 18.2 |
+----------------------------------------------------------------+