Toward self-correcting search engines: using underperforming queries to improve search

Search engines receive queries with a broad range of different search intents. However, they do not perform equally well for all queries. Understanding where search engines perform poorly is critical for improving their performance. In this paper, we present a method for automatically identifying poorly-performing query groups where a search engine may not meet searcher needs. This allows us to create coherent query clusters that help system design-ers generate actionable insights about necessary changes and helps learning-to-rank algorithms better learn relevance signals via spe-cialized rankers. The result is a framework capable of estimating dissatisfaction from Web search logs and learning to improve per-formance for dissatisfied queries. Through experimentation, we show that our method yields good quality groups that align with established retrieval performance metrics. We also show that we can significantly improve retrieval effectiveness via specialized rankers, and that coherent grouping of underperforming queries generated by our method is important in improving each group.