Image understanding for converting images into natural language text sentences

Abstract

The efficient processing, association and understanding of multimedia based events or multi-modal information is a very important research field with a great variety of applications, such as knowledge discovery, document understanding, human computer interaction, etc. A good approach to this important issue is the development of a common platform for converting different modalities (such as images, text, etc) into the same medium and associating them for efficient processing and understanding. Thus, this talk here presents the development of a methodology capable for automatically converting images into natural language (NL) text sentences using image processing-analysis methods and graphs with attributes for object recognition, and image understanding. Then it converts graph representations into NL text sentences. Moreover, it presents a methodology for transforming NL sentences into Graph representations and then into Stochastic Petri-nets (SPN) descriptions in order to offer a common model of representation of multimodal information and at the same time a way of associating “activities or changes” in image frames for events representation and interpretation. The selection of the SPN graph model is due to its capability for efficiently representing structural and functional knowledge where other models cannot. Simple illustrative examples are provided for proving the concept presented here.