Fusing Multimodal Human Expert Data to Uncover Hidden Semantics

Abstract

Problem solving in complex visual domains involves multiple levels of cognitive processing. Analyzing and representing these cognitive processes requires the elicitation and study of multimodal human data. We have developed methods for extracting experts' visual behaviors and verbal descriptions during medical image inspection. Now we address fusion of these data towards building a novel framework for organizing elements of expertise as a foundation for knowledge-dependent computational systems. In this paper, a multimodal graph-regularized non-negative matrix factorization approach is developed and used to fuse multimodal data collected during medical image inspection. Our experimental results on new data representation demonstrate the effectiveness of the proposed data fusion approach.