Comparing Rank Aggregation Methods based on Mallows Model

Abstract

Rank aggregation is the process of aggregating multiple base rankers into a single but more comprehensive ranker, which plays an important role in many domains such as recommender system, meta-search, database, genomics, etc. Works related to the comparison of rank aggregation methods all don’t have a suitable and general data generation mechanism to produce data with various characteristics and lack a more reasonable and effective algorithm evaluation performance index. Therefore, this paper presents a general data generation mechanism based on Mallows model to produce synthetic controllable datasets, uses generalized Kendall rank correlation coefficient and rank-biased overlap to evaluate and compare the performance of two kinds of methods under different settings. Besides, we also consider the comparison between indices and the impact of data characteristics on the algorithms. This paper may be helpful to researchers and decision-makers from multiple domains.