Why Web-Based Pseudo Relevance Feedback Systems Fail

Abstract

We review pseudo-relevance feedback as a mechanism for expanding short texts. Where short texts exhibit evolving concepts, topics and other characteristics, Web-based feedback systems were touted as the most ideal way of enriching the feature space of short texts. However, we note from a recent implementation of a Web-based pseudo-relevance feedback that it would only perform well under clinical situations. Further improvements to address fundamental noise in Web documents did not show significant improvements leading us to conclude that relevance feedback using Web documents directly are unsuitable for real-world conditions. In this paper, we present Eddi, which is a recent system that provides an exemplar of a typical pseudo-relevance feedback system. We first show the conditions in which Eddi will work and then discuss the situations where it would fail. We then present the variations to Eddi from our attempt to improve the robustness of Eddi's algorithm when dealing with complex Web documents. We then present the results from all variations to show the lack of robustness for pseudo-relevance feedback with Web documents.