文章目录

1. 描述2. 语法3. 参数4. 例子4.1 基础用法4.2 查看实时GPU性能4.3 限制GPU功率4.4 查看显存使用情况4.5 列出每个进程的GPU使用情况4.6 杀死占用GPU的进程4.7 导出GPU使用日志4.8 查询更多特定信息4.9 查看支持的GPU工作模式

1. 描述

<code>nvidia-smi（NVIDIA System Management Interface）是 NVIDIA 提供的命令行工具，用于监控和管理 NVIDIA GPU 的状态和性能。通过nvidia-smi，你可以查看 GPU 的使用情况、温度、功率、显存占用、驱动版本等信息。

2. 语法

基础用法：

nvidia-smi

详细描述：

nvidia-smi [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...

-h, --help Print usage information and exit.

LIST OPTIONS:

-L, --list-gpus Display a list of GPUs connected to the system.

-B, --list-excluded-gpus Display a list of excluded GPUs in the system.

SUMMARY OPTIONS:

<no arguments> Show a summary of GPUs connected to the system.

[plus any of]

-i, --id= Target a specific GPU.

-f, --filename= Log to a specified file, rather than to stdout.

-l, --loop= Probe until Ctrl+C at specified second interval.

QUERY OPTIONS:

-q, --query Display GPU or Unit info.

[plus any of]

-u, --unit Show unit, rather than GPU, attributes.

-i, --id= Target a specific GPU or Unit.

-f, --filename= Log to a specified file, rather than to stdout.

-x, --xml-format Produce XML output.

--dtd When showing xml output, embed DTD.

-d, --display= Display only selected information: MEMORY,

UTILIZATION, ECC, TEMPERATURE, POWER, CLOCK,

COMPUTE, PIDS, PERFORMANCE, SUPPORTED_CLOCKS,

PAGE_RETIREMENT, ACCOUNTING, ENCODER_STATS,

SUPPORTED_GPU_TARGET_TEMP, VOLTAGE

FBC_STATS, ROW_REMAPPER, RESET_STATUS

Flags can be combined with comma e.g. ECC,POWER.

Sampling data with max/min/avg is also returned

for POWER, UTILIZATION and CLOCK display types.

Doesn't work with -u or -x flags.

-l, --loop= Probe until Ctrl+C at specified second interval.

-lms, --loop-ms= Probe until Ctrl+C at specified millisecond interval.

SELECTIVE QUERY OPTIONS:

Allows the caller to pass an explicit list of properties to query.

[one of]

--query-gpu Information about GPU.

Call --help-query-gpu for more info.

--query-supported-clocks List of supported clocks.

Call --help-query-supported-clocks for more info.

--query-compute-apps List of currently active compute processes.

Call --help-query-compute-apps for more info.

--query-accounted-apps List of accounted compute processes.

Call --help-query-accounted-apps for more info.

This query is not supported on vGPU host.

--query-retired-pages List of device memory pages that have been retired.

Call --help-query-retired-pages for more info.

--query-remapped-rows Information about remapped rows.

Call --help-query-remapped-rows for more info.

[mandatory]

--format= Comma separated list of format options:

csv - comma separated values (MANDATORY)

noheader - skip the first line with column headers

nounits - don't print units for numerical

values

[plus any of]

-i, --id= Target a specific GPU or Unit.

-f, --filename= Log to a specified file, rather than to stdout.

-l, --loop= Probe until Ctrl+C at specified second interval.

-lms, --loop-ms= Probe until Ctrl+C at specified millisecond interval.

DEVICE MODIFICATION OPTIONS:

[any one of]

-pm, --persistence-mode= Set persistence mode: 0/DISABLED, 1/ENABLED

-e, --ecc-config= Toggle ECC support: 0/DISABLED, 1/ENABLED

-p, --reset-ecc-errors= Reset ECC error counts: 0/VOLATILE, 1/AGGREGATE

-c, --compute-mode= Set MODE for compute applications:

0/DEFAULT, 1/EXCLUSIVE_THREAD (DEPRECATED),

2/PROHIBITED, 3/EXCLUSIVE_PROCESS

--gom= Set GPU Operation Mode:

0/ALL_ON, 1/COMPUTE, 2/LOW_DP

-r --gpu-reset Trigger reset of the GPU.

Can be used to reset the GPU HW state in situations

that would otherwise require a machine reboot.

Typically useful if a double bit ECC error has

occurred.

Reset operations are not guarenteed to work in

all cases and should be used with caution.

-vm --virt-mode= Switch GPU Virtualization Mode:

Sets GPU virtualization mode to 3/VGPU or 4/VSGA

Virtualization mode of a GPU can only be set when

it is running on a hypervisor.

-lgc --lock-gpu-clocks= Specifies <minGpuClock,maxGpuClock> clocks as a

pair (e.g. 1500,1500) that defines the range

of desired locked GPU clock speed in MHz.

Setting this will supercede application clocks

and take effect regardless if an app is running.

Input can also be a singular desired clock value

(e.g. <GpuClockValue>). Optionally, --mode can be

specified to indicate a special mode.

-m --mode= Specifies the mode for --locked-gpu-clocks.

Valid modes: 0, 1

-rgc --reset-gpu-clocks

Resets the Gpu clocks to the default values.

-lmc --lock-memory-clocks= Specifies <minMemClock,maxMemClock> clocks as a

pair (e.g. 5100,5100) that defines the range

of desired locked Memory clock speed in MHz.

Input can also be a singular desired clock value

(e.g. <MemClockValue>).

-rmc --reset-memory-clocks

Resets the Memory clocks to the default values.

-lmcd --lock-memory-clocks-deferred=

Specifies memClock clock to lock. This limit is

applied the next time GPU is initialized.

This is guaranteed by unloading and reloading the kernel module.

Requires root.

-rmcd --reset-memory-clocks-deferred

Resets the deferred Memory clocks applied.

-ac --applications-clocks= Specifies <memory,graphics> clocks as a

pair (e.g. 2000,800) that defines GPU's

speed in MHz while running applications on a GPU.

-rac --reset-applications-clocks

Resets the applications clocks to the default values.

-pl --power-limit= Specifies maximum power management limit in watts.

Takes an optional argument --scope.

-sc --scope= Specifies the device type for --scope: 0/GPU, 1/TOTAL_MODULE (Grace Hopper Only)

-cc --cuda-clocks= Overrides or restores default CUDA clocks.

In override mode, GPU clocks higher frequencies when running CUDA applications.

Only on supported devices starting from the Volta series.

Requires administrator privileges.

0/RESTORE_DEFAULT, 1/OVERRIDE

-am --accounting-mode= Enable or disable Accounting Mode: 0/DISABLED, 1/ENABLED

-caa --clear-accounted-apps

Clears all the accounted PIDs in the buffer.

--auto-boost-default= Set the default auto boost policy to 0/DISABLED

or 1/ENABLED, enforcing the change only after the

last boost client has exited.

--auto-boost-permission=

Allow non-admin/root control over auto boost mode:

0/UNRESTRICTED, 1/RESTRICTED

-mig --multi-instance-gpu= Enable or disable Multi Instance GPU: 0/DISABLED, 1/ENABLED

Requires root.

-gtt --gpu-target-temp= Set GPU Target Temperature for a GPU in degree celsius.

Requires administrator privileges

[plus optional]

-i, --id= Target a specific GPU.

-eow, --error-on-warning Return a non-zero error for warnings.

UNIT MODIFICATION OPTIONS:

-t, --toggle-led= Set Unit LED state: 0/GREEN, 1/AMBER

[plus optional]

-i, --id= Target a specific Unit.

SHOW DTD OPTIONS:

--dtd Print device DTD and exit.

[plus optional]

-f, --filename= Log to a specified file, rather than to stdout.

-u, --unit Show unit, rather than device, DTD.

--debug= Log encrypted debug information to a specified file.

Device Monitoring:

dmon Displays device stats in scrolling format.

"nvidia-smi dmon -h" for more information.

daemon Runs in background and monitor devices as a daemon process.

This is an experimental feature. Not supported on Windows baremetal

"nvidia-smi daemon -h" for more information.

replay Used to replay/extract the persistent stats generated by daemon.

This is an experimental feature.

"nvidia-smi replay -h" for more information.

Process Monitoring:

pmon Displays process stats in scrolling format.

"nvidia-smi pmon -h" for more information.

TOPOLOGY:

topo Displays device/system topology. "nvidia-smi topo -h" for more information.

DRAIN STATES:

drain Displays/modifies GPU drain states for power idling. "nvidia-smi drain -h" for more information.

NVLINK:

nvlink Displays device nvlink information. "nvidia-smi nvlink -h" for more information.

C2C:

c2c Displays device C2C information. "nvidia-smi c2c -h" for more information.

CLOCKS:

clocks Control and query clock information. "nvidia-smi clocks -h" for more information.

ENCODER SESSIONS:

encodersessions Displays device encoder sessions information. "nvidia-smi encodersessions -h" for more information.

FBC SESSIONS:

fbcsessions Displays device FBC sessions information. "nvidia-smi fbcsessions -h" for more information.

GRID vGPU:

vgpu Displays vGPU information. "nvidia-smi vgpu -h" for more information.

MIG:

mig Provides controls for MIG management. "nvidia-smi mig -h" for more information.

COMPUTE POLICY:

compute-policy Control and query compute policies. "nvidia-smi compute-policy -h" for more information.

BOOST SLIDER:

boost-slider Control and query boost sliders. "nvidia-smi boost-slider -h" for more information.

POWER HINT: power-hint Estimates GPU power usage. "nvidia-smi power-hint -h" for more information.

BASE CLOCKS: base-clocks Query GPU base clocks. "nvidia-smi base-clocks -h" for more information.

CONFIDENTIAL COMPUTE:

conf-compute Control and query confidential compute. "nvidia-smi conf-compute -h" for more information.

GPU PERFORMANCE MONITORING:

gpm Control and query GPU performance monitoring unit. "nvidia-smi gpm -h" for more information.

Please see the nvidia-smi(1) manual page for more detailed information.

3. 参数

常见选项：

-q：显示详细信息。-d：显示特定信息（如性能状态、功率等）。-f：输出到文件。-i：指定GPU ID。-pl：设置功率限制。

4. 例子

4.1 基础用法

nvidia-smi

执行该命令会输出GPU的基本信息，包括：

驱动版本CUDA版本每个GPU的使用情况（温度、功率、显存占用、GPU利用率等）正在使用GPU的进程列表

4.2 查看实时GPU性能

使用 <code>-l 参数可以每隔一段时间输出一次 GPU 信息：

nvidia-smi -l 2

上面的命令每2秒输出一次 GPU 状态，直到你停止（Ctrl+C）。

4.3 限制GPU功率

有时为了节省电力或控制功耗，可以限制 GPU 的功率。假设你有一个功率范围为100W-250W的 GPU，可以将其限制为150W：

nvidia-smi -pl 150

4.4 查看显存使用情况

如果你只想查看当前GPU的显存使用情况，可以使用：

nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv

这个命令将以CSV格式输出每块GPU的总显存、已用显存和可用显存。

4.5 列出每个进程的GPU使用情况

想要查看每个进程使用的显存、GPU 资源，可以运行：

<code>nvidia-smi pmon -i 0

其中，-i 0 表示第0号 GPU，你可以根据需求选择其他 GPU。

4.6 杀死占用GPU的进程

你可以通过进程 ID（PID）找到占用显存或 GPU 资源的进程，然后使用 kill 命令终止它。例如，如果你看到进程 ID 为12345，可以这样终止它：

kill -9 12345

4.7 导出GPU使用日志

可以通过 nvidia-smi 将 GPU 信息输出到文件中，便于分析：

nvidia-smi -l 1 -f gpu_usage.log

这会将每秒的 GPU 使用情况记录到 gpu_usage.log 文件中。

4.8 查询更多特定信息

可以使用 --query-gpu 参数查询特定的信息。例如：

复制代码

nvidia-smi --query-gpu=name,temperature.gpu,utilization.gpu,memory.used --format=csv

这会以CSV格式输出GPU的名称、温度、利用率和已用显存。

4.9 查看支持的GPU工作模式

GPU 可以在不同的“性能模式”（如 P0 到 P12）下运行。可以通过 nvidia-smi 设置或查看当前的模式：

nvidia-smi --query-gpu=power.draw --format=csv