Integrating vision processing and natural language processing with a clinical application

Abstract

It is an interesting observation that humans are able to combine information from ostensibly disparate sources in order to better comprehend the world around them. For example, the sense information from the eyes and the ears are combined at every instant-tasks such as road crossing would be far more difficult without such integration. Such combinations are nonlinear in nature, so examining each sense in isolation will not shed light on how they combine within the cognitive frame. We are interested in the way humans combine their visual sense with natural language. This is an important area that appears to be little researched to date, despite the large amount of literature in both vision processing and natural language processing. In response to the nonlinear nature of the combination we proceed by synthesising the phenomenon rather than analysing it. To advance our work we focus upon a particular application that not only makes such a combination feasible but has a practical use in the field of medicine.