I wanted a nice way to simply see whatever statistics I could gather from my GPU — ye olde PNY XLR8 1070 slightly-pre-overclocked model, which I bought used and which has been surprisingly solid so far — and for some reason the information I want to see isn't in the default output from the nvidia-smi command included with the nvidia driver, and I don't get anything in lm-sensors. Noodling around a bit led me to the idea of using the --query-gpu option, which lets you pull any specific information you want, as long as you are willing to deal with it in CSV format. I was, and here we are.
If you only want the values, then you can suppress the header line and get just the data easily enough, for example:
$ nvidia-smi --query-gpu=memory.total,memory.reserved,memory.used,memory.free \
--format=csv,noheader
8192 MiB, 86 MiB, 733 MiB, 7372 MiB
You can suppress the units, too, with the nounits option to format. However, I wanted a tool that would show the labels as well as the values. I wound up writing a perl script to solve the problem. Some sample output:
timestamp: 2023/04/23 10:22:07.081
name: NVIDIA GeForce GTX 1070
utilization.gpu [%]: 1 %
utilization.memory [%]: 1 %
uuid: GPU-e3b77e2b-d4ae-d086-099e-c0f706a530f0
driver_version: 525.89.02
vbios_version: 86.04.50.00.E1
inforom.img: G001.0000.01.04
inforom.oem: 1.1
memory.total [MiB]: 8192 MiB
memory.reserved [MiB]: 86 MiB
memory.used [MiB]: 939 MiB
memory.free [MiB]: 7166 MiB
fan.speed [%]: 0 %
temperature.gpu: 58
compute_mode: Default
compute_cap: 6.1
encoder.stats.sessionCount: 0
encoder.stats.averageFps: 0
encoder.stats.averageLatency: 0
power.management: Enabled
pstate: P0
power.draw [W]: 35.23 W
power.limit [W]: 151.00 W
enforced.power.limit [W]: 151.00 W
power.default_limit [W]: 151.00 W
power.max_limit [W]: 170.00 W
clocks.current.graphics [MHz]: 1518 MHz
clocks.max.graphics [MHz]: 1923 MHz
clocks.current.sm [MHz]: 1518 MHz
clocks.max.sm [MHz]: 1923 MHz
clocks.current.video [MHz]: 1366 MHz
clocks.current.memory [MHz]: 4006 MHz
clocks.max.memory [MHz]: 4004 MHz
clocks_throttle_reasons.gpu_idle: Active
clocks_throttle_reasons.applications_clocks_setting: Not Active
clocks_throttle_reasons.sw_power_cap: Not Active
clocks_throttle_reasons.hw_slowdown: Not Active
clocks_throttle_reasons.hw_thermal_slowdown: Not Active
clocks_throttle_reasons.hw_power_brake_slowdown: Not Active
clocks_throttle_reasons.sw_thermal_slowdown: Not Active
clocks_throttle_reasons.sync_boost: Not Active
pcie.link.gen.current: 3
pcie.link.gen.gpucurrent: 3
pcie.link.width.current: 16