An Improved Job Co-Allocation Strategy in Multiple HPC Clusters

Abstract

To more effectively use HPC clusters, co-allocating jobs across multiple clusters becomes an attractive possibility with the primary benefit being reduced turnaround time. This, ultimately, depends on the inter- cluster communication cost. In our previous research, we introduced a co-allocation strategy, MBAS, that made use of two threshold values to control allocation: one for control link saturation and another to control job splitting. In this paper, we examine the performance of MBAS. A simulation study concludes that assigning jobs with different priorities according to their communication patterns, and adjusting the threshold values for link saturation level control and chunk size control in splitting jobs, the MBAS co- allocation strategy can significantly improve both user' satisfaction (in terms of turn around time) and system resource utilization consistently, even for jobs having large communication requirements.