Storing and retrieving perceptual episodic memories for long-term manipulation tasks

With recent technological advances, robotic agents are increasingly capable of performing ever more sophisticated manipulation tasks. Perceptual capabilities of these robots need to be able to adapt to the wide variety of tasks they are to perform. Remembering what a robot has seen, what the rationale was behind the decisions it took or how it ended up understanding the world as it did, are important questions if we want perception capabilities that can scale towards real-world manipulation. We present a robotic perception system that generates perceptual episodic memories during the execution of a task. To allow easy retrieval of these memories we introduce an object and scene description language that serves as a layer of abstraction between the structure of the perception logs and the semantic interpretation of these. The description language can be used through a query interface to retrieve specific parts of the generated episodic memory. The purpose of the proposed system is two-fold: to enable on-line retrospection and specialized training of perception routines and to enable researchers to interactively explore perception results.