SetDroid: Detecting User-Configurable Setting Issues of Android Apps via Metamorphic Fuzzing

Android, the most popular mobile system, offers a number of app-independent, user-configurable settings (e.g., network, location and permission) for controlling the devices and the apps. However, apps may fail to properly adapt their behaviors when these settings are changed, and thus frustrate users. We name such issues as setting issues, which reside in the apps and are induced by the changes of settings. According to our investigation, the majority of setting issues are non-crash (logic) bugs, which however cannot be detected by existing automated app testing techniques due to the lack of test oracles. To this end, we designed and introduced, setting-wise metamorphic fuzzing, the first automated testing technique to overcome the oracle problem in detecting setting issues. Our key insight is that, in most cases, the app behaviors should keep consistent if a given setting is changed and later properly restored. We realized this technique as an automated GUI testing tool, SetDroid, and applied it on 26 popular, open-source Android apps. SetDroid successfully found 32 unique, previously-unknown setting issues in these apps. So far, 25 have been confirmed and 17 were already fixed. We further applied SetDroid on 4 commercial apps with billions of monthly active users and successfully detected 15 previously unknown setting issues, all of which have been confirmed and under fixing. The majority of all these bugs (37 out of 47) are non-crash bugs, which cannot be detected by any prior testing technique.