Freeze It If You Can: Challenges and Future Directions in Benchmarking Smartphone Performance

Abstract

Benchmarking the performance of mobile devices such as Android-based smartphones is important in understanding and comparing the performance of different devices. Performance benchmarking tools such as Antutu have been widely used in both academia and industry. However, one of the main difficulties when benchmarking smartphone performance is due to the fact that the performance cannot be measured accurately and steadily. This paper investigates the challenges on performance benchmarking for Android-based smartphones. We identify key factors affecting performance benchmarking, which include frequency scaling setting, temperature, running background services, etc. Experiments show that some of these factors may cause performance fluctuation by as high as 60%. We show preliminary results in controlling the performance benchmarking process to generate steady results, for example freezing a smartphone in a refrigerator could remove most of the fluctuations. Finally, we discuss the implications of our study and possible future research directions.