Moving to exascale computing and beyond requires not only more processing speed (FLOPS), but also better energy efficiency (FLOPS/watt). Machines with heterogeneous architectures, such as those using graphical processing units (GPUs) as accelerators, are quickly establishing themselves as a next-generation, more sustainable alternative to traditional CPU-based compute.

GPUs employ large numbers (~10 000) of low-power cores, and as such are highly parallel. To exploit this parallelism, the employed programming paradigms need to be different from those used for CPUs. This presents challenges: algorithms often need to be rewritten to minimize data transfer between the CPU and the GPU, and to operate in bursts to saturate the device. Combining GPU parallelism with other levels of parallelism (e.g. MPI and OpenMP) is also challenging. In scientific software in particular, it is also important that portability is not sacrificed in the pursuit of faster code.

We present a modern, portable, single-codebase, OpenACC-based port of ONETEP, a linear-scaling density functional theory (DFT) program. ONETEP exploits electronic nearsightedness by reformulating Kohn-Sham DFT in terms of a single-particle density matrix and using localized orbitals. In so doing, it achieves linear scaling with system size (in contrast to the cubic scaling of conventional DFT), while maintaining plane-wave accuracy. This allows ONETEP to be routinely used in calculations on tens of thousands of atoms.

We carefully ported key algorithms identified as computational bottlenecks: the calculation of electronic density, local potential integrals, the gradient of the energy with respect to the expansion coefficients of localized orbitals, and two-electron integrals in Hartree-Fock exchange.

We demonstrate how our port offers a typical speed-up of 8x-12x in a fair comparison between a 48-core compute node with and without an NVIDIA A100 GPU card, while maintaining portability. We discuss how this only becomes possible when care is taken to minimize data transfer between the host and the device, which often leads to non-trivial changes to existing core algorithms. The resultant GPU port of ONETEP enables single-point energy calculations for ~100 000 atoms with plane-wave accuracy on relatively modest resources, significantly reducing time to science, enabling calculations in challenging problems that require complex models, such as fuel cells, batteries, or drug optimization.

Chris-Kriton Skylaris

Appearances