Multimedia Retrieval in Mixed Reality: Leveraging Live Queries for Immersive Experiences

Abstract

Recent advancements in Mixed Reality (MR) technology and the exponential growth of multimedia data production have led to the emergence of innovative approaches for efficient content retrieval. This paper introduces Mixed Reality Multimedia Retrieval ((MR)2), a groundbreaking concept at the convergence of MR and multimedia retrieval. At its core, (MR)2 leverages MR’s transformative capabilities with an innovative live query option, allowing users to initiate queries intuitively through real-world object interactions. By autonomously generating queries based on object recognition in the user’s field of view, (MR)2 facilitates the retrieval of similar multimedia content from a connected database. The technical backbone of the (MR)2 framework includes object detection (YOLOv8), semantic similarity search (CLIP), and data management (Cottontail DB). Our research redefines user interactions with multimedia databases, seamlessly bridging the physical and digital domains. A successful iOS prototype application demonstrates promising results, paving the way for immersive and context-aware multimedia retrieval in the MR era.