Active Perception based on Energy Minimization in Multimodal Human-robot Interaction

Humans use various types of modalities to express own internal states. If a robot interacting with humans can pay attention to limited signals, it should select more informative ones to estimate the partners' states. We propose an active perception method that controls the robot's attention based on an energy minimization criterion. An energy-based model, which has learned to estimate the latent state from sensory signals, calculates energy values corresponding to occurrence probabilities of the signals; The lower the energy is, the higher the likelihood of them. Our method therefore selects the modality that provides the lowest expectation energy among available ones to exploit more frequent experiences. We employed a multimodal deep belief network to represent relationships between humans' states and expressions. Our method demonstrated better performance for the modality selection than other methods in a task of emotion estimation. We discuss the potential of our method to advance human-robot interaction.