COMEX: A Multi-task Benchmark for Knowledge-grounded COnversational Media EXploration

Abstract

Open-domain conversational interaction with news, podcasts, and other types of heterogeneous content remains an open challenge. Interactive agents must support information access in a way that is fair, impartial, and true to the content and knowledge discussed. To facilitate this, systems building on interactive retrieval from knowledge-grounded media are a controllable and known base for experimentation. A conversational media agent should retrieve relevant content, understand key concepts in the content through grounding to a knowledge base, and enable exploration by offering to discuss a topic further or progress to describe related topics. In this work, we release a new multi-task benchmark on COnversational Media EXploration (COMEX) to measure knowledge-grounded conversational content exploration. It consists of a heterogeneous semantically annotated media corpus and topic-specific data for 1) entity Wikification and salience, 2) conversational content ranking on heterogeneous media content, 3) background link ranking, and 4) background linking explanation. We develop COMEX with judgments and conversational interactions developed in partnership with professional editorial staff from the BBC. We study the behavior of state-of-the-art systems, with the results demonstrating significant headroom on all tasks.