The P-Mesh-a commodity-based scalable network architecture for clusters

We designed a new network architecture, the P-Mesh, which combines the scalability and fault resilience of a torus with the performance of a switch. We compare the scalability, performance and cost of the hub, switch, torus, tree and P-Mesh architectures. The latter three are capable of scaling to thousands of nodes. However, the torus has severe performance limitations with that many processors. The tree and P-Mesh have similar latency, bandwidth and mid-bisection bandwidth, but the P-Mesh outperforms the switch architecture (a lower bound for tree performance) on 16-node NAS parallel benchmark tests by up to 23%, and costs 40% less. Furthermore, the P-Mesh has better fault resilience characteristics. The P-Mesh architecture trades increased management overhead for lower cost, and is a good bridging technology, while the price of tree uplinks is expensive.