A Massively Parallel Reservoir Simulator on the GPU Architecture

Reservoir simulation computational costs have been continuously growing due to high-resolution reservoir characterization, increasing model complexity, and uncertainty analysis workflows. Reducing simulation costs by upscaling is often necessary for operational requirements. Fast evolving HPC technologies offer opportunities to reduce cost without compromising fidelity. This work presents a novel in-house massively parallel full-physics reservoir simulator running on the emerging GPU architecture. Almost all the simulation kernels have been designed and implemented to honor the GPU SIMD programming paradigm. These kernels include physical property calculations, phase equilibrium computations, Jacobian construction, linear and nonlinear solvers, and wells. Novel techniques are devised in various kernels to expose enough parallelism to ensure that the control and data-flow patterns are well suited for the GPU environment. Mixed-precision computation is also employed when appropriate (e.g., in derivative calculation) to reduce computational costs without compromising the solution accuracy. The GPU implementation of the simulator is tested and benchmarked using various reservoir models, ranging from the synthetic SPE10 Benchmark (Christie & Blunt, 2001) to several industrial-scale models. These real field models range in size from tens of millions of cells to more than billion cells with black-oil and multicomponent compositional fluid. The GPU simulator is benchmarked on the IBM AC922 massively parallel architecture having tens of NVidia Volta V100 GPUs. To compare performance with CPU architectures, an optimized CPU implementation of the simulator is benchmarked on the IBM AC922 CPUs and on a cluster consisting of thousands of Intel's Haswell-EP Xeon® CPU E5-2680 v3. Detailed analysis of several numerical experiments comparing the simulator performance on the GPU and the CPU architectures is presented. In almost all of the cases, the analysis shows that the use of hardware acceleration offers substantial benefits in terms of wall time and power consumption. This novel in-house full-physics, black-oil and compositional reservoir simulator employs several novel techniques in various simulation kernels to ensure full utilization of the GPU resources. Detailed analysis is presented to highlight the simulator performance in terms of runtime reduction, parallel scalability and power savings.