An MPI-Stream Hybrid Programming Model for Computational Clusters

The MPI programming model hides network type and topology from developers, but also allows them to seamlessly distribute a computational job across multiple cores in both an intra and inter node fashion. This provides for high locality performance when the cores are either on the same node or on nodes closely connected by the same network type. The streaming model splits a computational job into a linear chain of decoupled units. This decoupling allows the placement of job units on optimal nodes according to network topology. Furthermore, the links between these units can be of varying protocols when the application is distributed across a heterogeneous network. In this paper we study how to integrate the MPI and Stream programming models in order to exploit network locality and topology. We present a hybrid MPI-Stream framework that aims to take advantage of each model's strengths. We test our framework with a financial application. This application simulates an electronic market for a single financial instrument. A stream of buy and sell orders is fed into a price matching engine. The matching engine creates a stream of order confirmations, trade confirmations, and quotes based on its attempts to match buyers with sellers. Our results show that the hybrid MPI-Stream framework can deliver a 32% performance improvement at certain order transmission rates.