Are Your GPU Atomics Secretly Contending?

GPU applications use atomic operations to coordinate data
access in highly parallel code. However, relying on previous
experiences and due to limited documentation, program-
mers resort to guidelines instead of concrete metrics to limit
potential performance influences.
In this paper, we introduce a GPU memory-subsystem
microbenchmark suite for analyzing GPU atomic operations.
Based on the benchmark results, we discuss two particu-
lar guidelines, namely: “use only one thread per warp to
access an atomic” and “place two atomic variables on differ-
ent cache lines to avoid contention.” We demonstrate where
these guidelines are effective and where actual hardware
behavior diverges.