To view all processes using GPU memory in Linux, you must use the NVIDIA System Management Interface (nvidia-smi) with specific query flags. The most direct method is running
nvidia-smi --query-compute-apps=pid,used_memory to list active compute contexts and their memory consumption. This command filters out idle drivers and shows only applications actively allocating VRAM, allowing for precise process management and troubleshooting of memory leaks or high-load tasks.
Prerequisites and Driver Verification
Before attempting to monitor GPU memory, ensure that the proprietary NVIDIA driver is installed and the kernel module is loaded correctly. Open-source Nouveau drivers often lack the necessary persistence mode support required for accurate process tracking. You can verify the installation by checking if the
nvidia-smi binary exists in your system path. Run the following command to check for the tool's availability:
nvidia-smi -L
If this command returns a list of GPUs with their names and UUIDs, the driver is functioning. If it fails or returns an error regarding missing libraries, reinstall the NVIDIA driver package appropriate for your distribution. For Ubuntu-based systems, this typically involves installing the
nvidia-driver-XXX meta-package via the package manager. Without a properly loaded kernel module, nvidia-smi cannot communicate with the hardware to retrieve process statistics.
Using Query-Compute-Apps for Process Listing
The primary tool for identifying which processes are consuming GPU memory is the
--query-compute-apps flag within
nvidia-smi. This flag specifically targets compute applications, excluding display servers or other non-compute contexts unless they are also performing calculations. The standard syntax requires specifying the columns you wish to retrieve. The two most critical columns are
pid for the process ID and
used_memory for the amount of VRAM consumed.
Execute the following command to see a clean list of processes:
nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader,nounits
This output will resemble a simple table where each line contains a PID and its corresponding memory usage in megabytes. The
--format=csv option ensures machine-readable output, while
noheader removes column titles for easier parsing. The
nounits flag prevents the addition of "MiB" text, making it simpler to script against if needed. This approach directly answers the need to identify heavy consumers without wading through verbose default outputs.
Monitoring with Dynamic Monitoring (dmon)
For real-time observation rather than a static snapshot, the
dmon subcommand provides dynamic monitoring capabilities. This tool updates the screen periodically, showing changes in memory usage, temperature, and utilization. It is particularly useful when debugging applications that leak memory over time or fluctuate significantly in demand.
Run the dynamic monitor with the following arguments:
nvidia-smi dmon -i 0 -c 1
The
-i 0 flag sets the update interval to zero seconds, meaning it refreshes as fast as possible, while
-c 1 limits the number of iterations to one if used in a script, or you can omit
-c for continuous monitoring until interrupted with Ctrl+C. The output includes columns for PID, name, type, memory usage, and utilization percentage. This view helps correlate spikes in memory usage with specific application behaviors.
Interpreting Memory Usage Types
It is important to distinguish between different types of GPU memory usage. Compute apps report memory allocated for kernels and data structures. However, some memory may be reserved by the driver for context switching or graphics operations. If
nvidia-smi reports high memory usage but no compute apps are listed, the memory might be held by the X server or Wayland compositor. In such cases, checking the
--query-gpu options can provide broader context on total available versus free memory.
Scripting and Automation
Integrating these commands into shell scripts allows for automated alerts or cleanup routines. You can parse the CSV output from
--query-compute-apps using standard text processing tools like
awk or
grep. For example, to find any process using more than 5000 MiB of memory, you could construct a pipeline that filters the numeric value.
Here is a basic example of how to extract PIDs exceeding a threshold:
nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader,nounits | awk -F',' '$2 > 5000 {print $1}'
This snippet prints only the PIDs of processes consuming excessive VRAM. Such automation is valuable in server environments where resources must be strictly managed. You can combine this with notification services to alert administrators of potential issues before they cause system instability.
Managing High-Memory Processes
Once you have identified the offending processes, you may need to terminate them to free up resources. Use the standard Linux
kill command with the PID obtained from the previous steps. Always attempt a graceful termination first using
SIGTERM before resorting to
SIGKILL.
kill [PID]
If the process does not respond, force its removal with
kill -9 [PID]. After killing the process, verify that the memory has been released by running the query command again. Persistent memory leaks might require restarting the entire GPU driver stack, which can be done by reloading the kernel module or rebooting the system. For more detailed guidance on freeing memory, refer to our guide on
Freeing GPU Memory in Linux: Boosting Performance and Efficiency.
Frequently Asked Questions
Why does nvidia-smi show memory usage but no processes?
This occurs when the GPU is being used by the display server (X11 or Wayland) or a background service rather than a compute application. The
--query-compute-apps flag excludes these graphical contexts. To see all memory allocation, including graphics, use
--query-gpu=memory.used,memory.total instead.
How do I monitor GPU memory on AMD GPUs?
NVIDIA-specific tools like nvidia-smi do not work on AMD hardware. Instead, use
rocm-smi or
radeontop for AMD GPUs. These tools provide similar functionality for listing processes and monitoring VRAM usage on ROCm-enabled systems.
Can I limit GPU memory usage for a specific process?
Yes, you can set environment variables like
CUDA_VISIBLE_DEVICES to restrict access, but limiting memory size directly is handled at the application level. Some deep learning frameworks allow setting memory fraction limits. Alternatively, use cgroups or systemd resource controls to limit overall system memory, though this affects RAM, not VRAM specifically.
What causes sudden spikes in GPU memory usage?
Sudden spikes are often caused by large batch sizes in machine learning training, video encoding tasks, or rendering jobs. Check the logs of the active application to see if it is loading new datasets or textures. Reducing batch sizes or optimizing data pipelines can mitigate these spikes.
Is it safe to kill a process using GPU memory?
Killing a process frees the VRAM it holds, but it may crash the application abruptly. Ensure that saving state is possible or that data loss is acceptable. If the process is part of a critical service, coordinate with the application owner before terminating it to avoid data corruption or service interruption.