Continuous Usability Requirements Evaluation based on Runtime User Behavior Mining

Usability requirements have been widely recognized as an essential quality requirement for systems that interact with people. However, evaluating the satisfaction of usability requirements usually involves user interactions, which is intrusive and time-consuming. In this paper, we propose a novel framework for systematically and automatically evaluating the satisfaction of usability requirements at runtime. Specifically, a behavior-centric conceptual model is proposed to comprehensively characterize user behaviors. An analysis process is then proposed based on the conceptual model, which systematically refines high-level usability requirements into observable and measurable user behaviors in order to automatically evaluate their satisfaction. Moreover, we investigate and mine patterns of user behaviors, which further explain the results of the satisfaction analysis. We systematically design and conduct a case study to evaluate our proposed framework, the results of which show that our approach is able to identify most usability issues and precisely assess the satisfaction of participants’ usability requirements. Importantly, our approach enables continuous usability requirements evaluation without interfering with users, pragmatically contributing to trade-off analysis among quality requirements at runtime.