Optimized GPU usage in high energy physics applications

Machine learning (ML) applications, which have become quite common tools for
many high energy physics (HEP) applications, benefit significantly from GPU resources. To
fulfill the rapidly increasing demand for GPU resources, the efficient utilization of GPU clusters
is vital. The Karlsruhe Institute of Technology (KIT) provides a GPU cluster, accessible via
a traditional batch system as well as via grid compute elements. Because the exact hardware
needs of such applications heavily depend on the ML hyperparameters, a flexible resource setup
is necessary to utilize the available resources as efficiently as possible. Therefore, the multi-
instance GPU (MIG) feature of the NVIDIA A100 GPUs was studied. Several neural network
(NN) training scenarios performed on the GPU cluster at KIT are discussed to illustrate the
setup that has been used and possible performance gains.