TopK Ordering on Distributed Systems

Abstract

Ranking has wide range of applications like social choice\citeSC, recommendation systems\citeRS, web search\citeWS, crowd sourcing \citeCS etc. \textttTeraSort is a distributed algorithm, commonly used in systems like Hadoop MapReduce, for sorting large datasets. However, in most applications of interest we do not desire complete ordering of data, rather only a few items which have the highest ranks. In this paper we propose Coded Partial Sort to obtain partially sorted data from large datasets using distributed computing systems. We intend to find \texttttopK ordered elements of a dataset by optimally utilizing servers in distributed network.\\ Coded Partial Sort modifies conventional \textttTeraSort algorithm to remove data irrelevant for partial ordering and applies ideas of "coding" to improve run-time performance by significantly decreasing communication load of Uncoded Partial Sort\citeUs. We empirically evaluate the performance of tCoded and Uncoded Partial Sort on Amazon EC2 clusters for experimental settings of interest.