RepSys: Framework for Interactive Evaluation of Recommender Systems

Making recommender systems more transparent and auditable is crucial for the future adoption of these systems. Available tools typically present mostly errors of models aggregated over all test users, which is often insufficient to uncover hidden biases and problems. Moreover, the emphasis is primarily on the accuracy of recommendations but less on other important metrics, such as the diversity of recommended items, the extent of catalog coverage, or the opportunity to discover novel items at bestsellers’ expense. In this work, we propose RepSys, a framework for evaluating recommender systems. Our work offers a set of highly interactive approaches for investigating various scenario recommendations, analyzing a dataset, and evaluating distributions of various metrics that combine visualization techniques with existing offline evaluation methods. RepSys framework is available under an open-source license to other researchers.