Beyond accuracy measures: the effect of diversity, novelty and serendipity in recommender systems on user engagement

Abstract

The quality of recommender systems (RS) is typically measured by their predictive accuracy. There is an emerging understanding that RS must provide not just accuracy, but also usefulness and enhanced user engagement, where diversity, novelty, and serendipity have been identified as the most common quality features to improve the RS beyond accuracy measures. This research investigates how diversity, novelty and serendipity of the recommended items as well as user’s prosumer behavior affect user engagement dynamically. We formulate a dynamic panel data model using the data collected from NetEase Cloud Music, one of China’s largest music streaming platforms. The findings indicate that both novelty and serendipity of the recommended items have positive impact on user engagement while a more diversified recommendation list could hurt user engagement. Our findings also suggest being a prosumer who also creates videos instead of a pure consumer of music videos will make the user more engaged with the platform in the long run. In addition, our findings clarify the relationship between prosumer behavior and the impact of diversity, novelty and serendipity on user engagement. Being a prosumer alters the effect of diversity on user engagement from negative to positive. Also, creators are drawn to unpopular and unexpected videos as they serve as a source of inspiration for their creative endeavors. The findings of this study have substantial implications for music streaming platforms and other social media and e-commerce platforms to leverage long-term customer engagement through the improvement of recommender systems. For example, a targeted 90-2-20 rule can be implemented to balance the diversity, novelty and serendipity of the recommended items, which prioritizes the selection of 90% of recommended items from the user’s top 2 preferred genres, the remaining 10% from unrecommended genres, and includes 20% of unpopular items within each genre. To encourage the users to create contents, various means can be applied by the platforms such as bestowing a creator badge, offering reward cashback and subscription discounts.