Remotely Disabling and Enabling NVIDIA GPU using Software
In an administrator's day-to-day tasks, there might be a need to control the GPU devices in a remote machine. This article will cover the concepts, steps, and sample code to remotely trigger the disabling and enabling of NVIDIA GPU devices using software.
Why Disable/Enable NVIDIA GPU Remotely?
Managing GPU devices, especially in a remote machine, can help with various use cases like load balancing, allocating resources, troubleshooting, and maintenance. An admin can disable a GPU to free up resources or enable it back for specific applications or processes.
Understanding NVIDIA GPU Management
NVIDIA provides a set of APIs called the NVML (NVIDIA Management Library). This library can be used for monitoring and managing NVIDIA GPUs installed in a system programmatically. The library provides functions for querying GPU metrics and controlling specific device states like enabling, disabling, and resetting.
Prerequisites and Environment Setup
To set up and test the remote GPU management, make sure to prepare two machines:
- Target Machine: A system with an NVIDIA GPU installed, and the NVIDIA driver installed.
- Client Machine: A system from where you will trigger the remote GPU management. Make sure you can connect to the target machine using SSH or any other preferred method.
Follow the next steps to configure your machines:
- Install the required libraries for the client machine: For this article, Python will be used to demonstrate connecting and controlling the remote GPU. Install the necessary libraries using pip:
pip install pyssh paramiko
- Enable Passwordless SSH Login: To automate the process, configure passwordless SSH login between the client and target machines. Follow the guidelines in this article:
Python Sample for Remotely Disabling/Enabling NVIDIA GPU
This section will cover a Python script that can be used on the client machine to remotely trigger the enabling and disabling of the NVIDIA GPU present on a target machine.
Create a Python script called remote_gpu_manager.py with the following content:
import paramiko
import os
# Replace these variables with your target machine's IP and username
HOST = '192.168.1.100'
USERNAME = 'nvidia_admin'
def execute_command(command):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(HOST, username=USERNAME)
stdin, stdout, stderr = ssh.exec_command(command)
return stdout.read()
def enable_gpu():
command = 'nvidia-smi -i 0 -p 0 -g 0 --set-power-usage=max'
return execute_command(command)
def disable_gpu():
command = 'nvidia-smi -i 0 -p 0 -g 0 --sleep 30 --set-power-usage=0'
return execute_command(command)
if __name__ == '__main__':
choice = int(input("Enter 1 to enable or 2 to disable NVIDIA GPU: "))
if choice == 1:
enable_gpu()
elif choice == 2:
disable_gpu()
else:
print("Invalid Option")
Summary and References
In this article, we covered the basics of disabling and enabling a remote NVIDIA GPU using software. You can use the NVML library to manage your GPU devices. In addition, we presented a Python script that utilizes SSH to connect and control NVIDIA GPUs installed on a target machine.