Statistical information retrieval modelling: from the probability ranking principle to recent advances in diversity, portfolio theory, and beyond

Abstract

Statistical modelling of Information Retrieval (IR) systems is a key driving force in the development of the IR field. The goal of this tutorial is to provide a comprehensive and up-to-date introduction to statistical IR modelling. We take a fresh and systematic perspective from the viewpoint of portfolio theory of IR and risk management. A unified treatment and new insights will be given to reflect the recent developments of considering the ranked retrieval results as a whole. Recent research progress in diversification, risk management, and portfolio theory will be covered, in addition to classic methods such as Maron and Kuhns' Probabilistic Indexing, Robertson-Sparck Jones model (and the resulting BM25 formula) and language modelling approaches. The tutorial also reviews the resulting practical algorithms of risk-aware query expansion, diverse ranking, IR metric optimization as well as their performance evaluations. Practical IR applications such as web search, multimedia retrieval, and collaborative filtering are also introduced, as well as discussion of new opportunities for future research and applications that intersect among information retrieval, knowledge management, and databases.