Experimental Performance Evaluation Among Cloud Infrastructure Providers Under Different Load Levels

Abstract

Background: Performance testing can estimate the capacity of a web service under requests. Decision-making on the ideal cloud service infrastructures for deploying specific cloud applications is challenging. Goal: Investigation on the performance of cloud infrastructure providers under different load levels. Method: An experimental study evaluated Amazon, Azure, Google, and IBM cloud infrastructure providers in terms of performance under Infrastructure as a Service (IaaS) perspective. Results: The results indicated satisfactory performance in response time and latency among most providers when subjected to up to 300 simultaneous threads. However, this effect decreases as the number of threads increases, and Apdex index value and level of user's satisfaction are significantly reduced under different load levels, mainly for the IBM provider. Besides, the error rate rises substantially at 400 threads, with more critical results for IBM and Amazon providers. Conclusions: Providers show distinct differences across metrics, and the data collected during testings reinforced the potential particularities of cloud infrastructure services for the choice of a provider. Although preliminary, such results can support software companies in selecting a proper infrastructure provider according to particular requirements.