Caiipa: automated large-scale mobile app testing through contextual fuzzing

Scalable and comprehensive testing of mobile apps is extremely challenging. Every test input needs to be run with a variety of contexts, such as: device heterogeneity, wireless network speeds, locations, and unpredictable sensor inputs. The range of values for each context, e.g. location, can be very large. In this paper we present Caiipa, a cloud service for testing apps over an expanded mobile context space in a scalable way. It incorporates key techniques to make app testing more tractable, including a context test space prioritizer to quickly discover failure scenarios for each app. We have implemented Caiipa on a cluster of VMs and real devices that can each emulate various combinations of contexts for tablet and phone apps. We evaluate Caiipa by testing 265 commercially available mobile apps based on a comprehensive library of real-world conditions. Our results show that Caiipa leads to improvements of 11.1x and 8.4x in the number of crashes and performance bugs discovered compared to conventional UI-based automation (i.e., monkey-testing).