BioMDSE: A Multimodal Deep Learning-Based Search Engine Framework for Biofilm Documents Classifications

As biofilms research grows rapidly, a corpus of bibliographic literature (i.e., documents) is increasing at an incredible rate. Many researchers often need to inspect these large document collections, including (1) text, (2) images, and (3) captions, to understand underlying biological mechanisms and make a critical decision. However, researchers have great difficulty in exploring such ever-growing large datasets in labor-intensive processes. Thus, automation of such tasks is urgently required for the automatic identification or classification of a large volume of document collections. To address this problem, we present a multimodal deep learning-based approach to automatically classify documents for a specialized information retrieval technique based on biofilm images, captions, and texts, which is a major source of information for the classification of documents. Images, captions, and texts from biofilm documents are represented in a large vector space. Then, they are fed into convolutional neural networks (CNNs), to improve similarity matching and relevance. Our extensive experiments and analysis will take captions, texts, or images as unimodal models as inputs and concatenate them all into multimodal models. The trained models for this classification approach in turn help a search engine to precisely identify relevant and domain-specific documents from a large volume of document collections for further research direction in biofilm development.