Use nvidia-smi with the --query-gpu=memory.used,memory.total,index flags and parse the output to check if available memory exceeds a threshold. A bash loop retries every few seconds until the condition is met, ensuring your PyTorch script starts only when sufficient GPU resources are free.
Understanding GPU Memory Constraints
When running deep learning workloads, multiple processes often compete for limited Video RAM (VRAM). If you attempt to launch a new training job while another process holds significant memory, PyTorch will fail with an Out of Memory (OOM) error. This is particularly common in shared server environments or when debugging residual memory leaks from previous runs. To automate this process, you need a script that actively monitors the Graphics Processing Unit (GPU) state. The standard tool for this on Linux systems isnvidia-smi. However, parsing its output correctly is critical because the format can vary depending on whether you query specific metrics or use default table views. Many existing scripts fail because they assume a static column layout or ignore the specific GPU index being targeted.
For more context on how to inspect these resources manually, see our guide on viewing processes using GPU memory in Linux. Understanding which processes consume VRAM helps in diagnosing why memory might not be freeing up as expected.
Constructing the Monitoring Script
The core logic involves querying the current memory usage, calculating the available portion, and comparing it against a desired minimum. Below is a robust implementation that addresses common pitfalls such as variable scoping and output parsing. Create a file namedwait_for_gpu.sh and insert the following code:
#!/bin/bash
# Configuration
GPU_ID=${1:-0} # Default to GPU 0 if not specified
MIN_FREE_GB=${2:-8} # Minimum free GB required (default 8GB)
CHECK_INTERVAL=5 # Seconds between checks
MAX_RETRIES=100 # Safety limit to prevent infinite loops
echo "Waiting for GPU $GPU_ID to have at least ${MIN_FREE_GB}GB free..."
retries=0
while [ $retries -lt $MAX_RETRIES ]; do
# Query specific metrics: used memory, total memory, and index
# Using comma-separated query ensures predictable output order
output=$(nvidia-smi --query-gpu=memory.used,memory.total,index --format=csv,noheader,nounits -i $GPU_ID)
# Check if nvidia-smi returned valid data
if [ -z "$output" ]; then
echo "Error: Could not retrieve GPU info. Is the driver loaded?"
exit 1
fi
# Parse the CSV output
# Format: used,total,index
# Example: 1024,8192,0
used_mem=$(echo "$output" | cut -d',' -f1)
total_mem=$(echo "$output" | cut -d',' -f2)
# Calculate free memory in MB
free_mem_mb=$((total_mem - used_mem))
free_mem_gb=$(echo "scale=2; $free_mem_mb / 1024" | bc)
echo "Current Free Memory: ${free_mem_gb} GB"
# Check if free memory meets threshold
# Using bc for floating point comparison
if (( $(echo "$free_mem_gb >= $MIN_FREE_GB" | bc -l) )); then
echo "GPU $GPU_ID is ready! Starting training..."
break
fi
retries=$((retries + 1))
echo "Waiting ${CHECK_INTERVAL}s... (Attempt $retries/$MAX_RETRIES)"
sleep $CHECK_INTERVAL
done
if [ $retries -eq $MAX_RETRIES ]; then
echo "Timeout: GPU did not free up enough memory."
exit 1
fi
# Execute the target command passed as arguments
if [ $# -gt 2 ]; then
shift 2
exec "$@"
else
echo "No command provided to execute after wait."
exit 1
fi
Key Implementation Details
This script avoids several common bugs found in simpler implementations. First, it uses the-i $GPU_ID flag to explicitly target the correct GPU. Older scripts often rely on line numbers in multi-GPU outputs, which breaks if GPUs are indexed differently or if the query returns unexpected rows. By specifying the index, we ensure cut -d',' -f1 always refers to the memory of the intended device.
Second, the script uses bc for arithmetic. Bash does not support floating-point math natively. Calculating gigabytes from megabytes requires division, which bc handles precisely. Without bc, integer division would truncate values incorrectly, leading to false positives where low memory appears sufficient.
Third, the script accepts the training command as arguments. The shift 2 command removes the first two positional parameters (GPU ID and Memory Threshold), leaving the remaining arguments to be executed via exec. This allows you to pass environment variables or complex PyTorch commands directly without subshell overhead.
If you encounter issues with script execution permissions or path resolution, review our article on Linux RHEL 7 shell script execution issues for troubleshooting steps.
Usage Examples
Make the script executable:chmod +x wait_for_gpu.sh
./wait_for_gpu.sh 1 10 python train.py --epochs 100
train.py.