transformers/docs/source/en/gpu_selection.md

GPU selection

During distributed training, you can specify the number of GPUs to use and in what order. This can be useful when you have GPUs with different computing power and you want to use the faster GPU first. Or you could only use a subset of the available GPUs. The selection process works for both DistributedDataParallel and DataParallel. You don't need Accelerate or DeepSpeed integration.

This guide will show you how to select the number of GPUs to use and the order to use them in.

Number of GPUs

For example, if there are 4 GPUs and you only want to use the first 2, run the command below.

Use the --nproc_per_node to select how many GPUs to use.

torchrun --nproc_per_node=2  trainer-program.py ...

Use --num_processes to select how many GPUs to use.

accelerate launch --num_processes 2 trainer-program.py ...

Use --num_gpus to select how many GPUs to use.

deepspeed --num_gpus 2 trainer-program.py ...

Order of GPUs

To select specific GPUs to use and their order, configure the the CUDA_VISIBLE_DEVICES environment variable. It is easiest to set the environment variable in ~/bashrc or another startup config file. CUDA_VISIBLE_DEVICES is used to map which GPUs are used. For example, if there are 4 GPUs (0, 1, 2, 3) and you only want to run GPUs 0 and 2:

CUDA_VISIBLE_DEVICES=0,2 torchrun trainer-program.py ...

Only the 2 physical GPUs (0 and 2) are "visible" to PyTorch and these are mapped to cuda:0 and cuda:1 respectively. You can also reverse the order of the GPUs to use 2 first. The mapping becomes cuda:1 for GPU 0 and cuda:0 for GPU 2.

CUDA_VISIBLE_DEVICES=2,0 torchrun trainer-program.py ...

You can also set the CUDA_VISIBLE_DEVICES environment variable to an empty value to create an environment without GPUs.

CUDA_VISIBLE_DEVICES= python trainer-program.py ...

Warning

As with any environment variable, they can be exported instead of being added to the command line. However, this is not recommended because it can be confusing if you forget how the environment variable was set up and you end up using the wrong GPUs. Instead, it is common practice to set the environment variable for a specific training run on the same command line.

CUDA_DEVICE_ORDER is an alternative environment variable you can use to control how the GPUs are ordered. You can order according to the following.

1. PCIe bus IDs that matches the order of nvidia-smi and rocm-smi for NVIDIA and AMD GPUs respectively.

export CUDA_DEVICE_ORDER=PCI_BUS_ID

2. GPU compute ability.

export CUDA_DEVICE_ORDER=FASTEST_FIRST

The CUDA_DEVICE_ORDER is especially useful if your training setup consists of an older and newer GPU, where the older GPU appears first, but you cannot physically swap the cards to make the newer GPU appear first. In this case, set CUDA_DEVICE_ORDER=FASTEST_FIRST to always use the newer and faster GPU first (nvidia-smi or rocm-smi still reports the GPUs in their PCIe order). Or you could also set export CUDA_VISIBLE_DEVICES=1,0.