NVIDIA GPU sharing under Linux
At the time of writing, NVIDIA didn’t provide any tools for monitoring the usage of their boards. Although they were “looking into it”. This page describes the hackish solutions I provided to share boards in Toronto. When running jobs on CPUs it is most efficient to only run one job per CPU. On GPUs this rule seems more important, as system crashes result more often when multiple jobs are run at once(?).
I have written a Python script
implements a discretionary locking scheme: if everyone uses it, only one user
will use a given GPU ID at a time. If an ID lock is obtained in the default way,
the lock is automatically freed when the parent process ends (even if it
crashes) or on a system reboot. It is also possible to get a lock that will
persist and that must be manually freed.
Toronto ML users: the command to alias is
To use it as a Python module, symlink both
/u/murray/bin/run_on_me_or_pid_quit into a directory in your
Edinburgh ML users: the command to alias is
/disk/scratch/imurray2/gpu_lock/gpu_lock.py. To use it as a Python module, symlink both
/disk/scratch/imurray2/gpu_lock/run_on_me_or_pid_quit into a directory in your
Run the script with no arguments to see the locks currently assigned and for more information on using the script. It should be easy to use. In a shell script do:
Then tell your GPU-using program to use
$ID. From within a Python program do:
import gpu_lock id = gpu_lock.obtain_lock_id()
From within Matlab do:
id = obtain_gpu_lock_id();
Generically, in a programming language with a system command that runs a sub-shell do:
id = str2int(system("exec gpu_lock.py --id"));
More examples are given in the directory. For all of
the above methods of locking an id, the lock will be freed when the calling
program finishes. An ID of -1 is returned if none are free. Run
gpu_lock.py with no arguments for help on how to get a persistent lock
and freeing it manually. But this is not recommended.
The script doesn't really know anything about GPUs other than the number of boards available. It's just a way of assigning integers on a first-come first-served basis.
Portability: The number of boards is found in a linux-specific way (looking
at the number of
/dev/nvidia[0-9]* devices) as is process monitoring
/proc) for lock freeing. Some tweaking is probably required
for OSX. The symlink-based locking mechanism should work on any POSIX system
(Linux and OSX, not Windows).
Our first approach:
Note: this section describes a tool we no longer really use as it doesn't work properly, but it may be of some use.
It would be nice to see if a GPU ID is actually being used, rather than just
locked. The first script I wrote was gpu_usage. It uses
a setuid wrapper to
/usr/bin/lsof to see which users were using
/dev/nvidia[0-9]* devices as memory-mapped files. This used to work, but now
(for reasons unknown, but presumably after something was upgraded) a user using
just one of the devices for real computation appears to be using all of the
devices. So the script currently over-reports usage.
In some ways
gpu_lock is better anyway, despite not knowing about
real usage. It ensures that no two users use the same ID (as long as everyone
uses the locking system). With
gpu_usage we had problems with jobs
simultaneously requesting an ID.