Multimodal Interactive Machine Learning for User Understanding

Abstract

Designing intelligent computer interfaces requires human intelligence, which can be captured through multimodal sensors during human-computer interactions. These data modalities may involve users' language, vision, and body signals, which shed light on different aspects of human cognition and behaviors. I propose to integrate multimodal data to more effectively understand users during interactions. Since users' manipulation of big data (e.g., texts, images, videos) through interfaces can be computationally intensive, an interactive machine learning framework will be constructed in an unsupervised manner.