Image Synthesis from Themes Captured in Poems using Latent Diffusion Models

Abstract

Due to the presence of complex literary devices such as metaphors and imagery, poetry can be difficult to comprehend. This is especially true for literary works of classical poets like Kaalidasa that employ intricate, often conflicting themes which tend to be particularly tedious to interpret and decipher. The beauty of these works of art tends to get lost in translation. A visual representation in the form of images corresponding to the various themes in the poetry, greatly aids in providing a clearer understanding of the meaning and imagery described. The main aim here is to make classical poetry more accessible by generating detailed images that capture and depict the metaphors and themes used in various works of literature. The core task in this paper is to employ novel machine learning (NLP) techniques to detect and extract the central themes and keywords from the poems that encapsulate the essence of the literary work. This is done using transformer models fine-tuned specifically on a summarization dataset, that generate an abstractive summary of the segment of input text. Maintaining context while doing so is essential to the accuracy of the images being generated. Further, this summary is then provided as an input to a Latent Diffusion Model to generate detailed images corresponding to the poetry. The goal of the project is to make it easier to consume and enjoy classical works of literature by providing additional context and information in the form of images complementing the poetry.