Computational Methods for Integrating Vision and Language

Abstract "This is clearly the most comprehensive and thoughtful compendium of knowledge on language/vision integration out there, and I'm sure it will be a valuable resources to many researchers and instructors." - Sven Dickinson, Series Editor (University of Toronto) Modeling data from visual and linguistic modalities together creates opportunities for better understanding of both, and supports many useful applications. Examples of dual visual-linguistic data includes images with keywords, video with narrative, and figures in documents. We consider two key task-driven themes: translating from one modality to another (e.g., inferring annotations for images) and understanding the data using all modalities, where one modality can help disambiguate information in another. The multiple modalities can either be essentially semantically redundant (e.g., keywords provided by a person looking at the image), or largely complementary (e.g., meta data such as the camera used). Redundancy and complementarity are two ...