Performance of MPI Codes Written in Python with NumPy and mpi4py

Abstract

Python is an interpreted language that has become more commonly used within HPC applications. Python benefits from the ability to write extension modules in C, which can further use optimized libraries that have been written in other compiled languages. For HPC users, two of the most common extensions are NumPy and mpi4py. It is possible to write a full computational kernel in a compiled language and then build that kernel into an extension module. However, this process requires not only the kernel be written in the compiled language, but also the interface between the kernel and Python be implemented. If possible, it would be preferable to achieve similar performance by writing the code directly in Python using readily available performant modules. In this work the performance differences between compiled codes and codes written using Python3 and commonly available modules, most notably NumPy and mpi4py, are investigated. Additionally, the performance of an open source Python stack is compared to the recently announced Intel Python3 distribution.