How to See Smells: Extracting Olfactory References from Artworks

1 PROBLEM Although being an essential part of how we experience the world, smell is severely undervalued in the context of cultural heritage. The Odeuropa project aims at preserving and recreating the olfactory heritage of Europe. State-of-the-art methods of artificial intelligence are applied to large corpora of visual and textual data ranging from the 16th to 20th century of European history to extract olfactory references. Creating an ontology of smells, this information is stored in the “European Olfactory Knowledge Graph (EOKG)” following standards of the semantic web. My Ph.D. addresses the visual extraction part of the project. We will create a taxonomy of visual smell references and acquire a large corpus of artworks from various early modern European digital collections. Using computer vision techniques, we will implement a pipeline for the combined recognition of olfactory objects, poses, and iconographies and annotate the images from our image corpus accordingly. Following these steps, we will address the following research questions: (i)What visual representations of smell exist in European 16th to 20th century works of art and how can these be represented in the EOKG as an ontology shared with the other work packages of the Odeuropa project? (ii)Whichmachine-learning techniques exist for the automated extraction of olfactory references in the visual arts? Particularly, which techniques are suited to cope with the domain shift problem when applying computer vision techniques to our field of research? (iii) How do the identified techniques perform in terms of established evaluation metrics? Which ones work best for the extraction of olfactory references? Both the preservation of olfactory heritage [3] and the application of machine learning (ML) to cultural heritage [1] have been addressed before. However, in most cases machine learning algorithms are treated as “black boxes” and their application does not contribute back to ML [4]. Computer vision techniques like object detection and pose estimation have successfully been applied to the domain of visual arts ([8], [2]) but have not achieved performance comparable to their application in the photographic domain. One reason for the success of computer vision on photographs is the availability of huge labeled datasets like ImageNet [10]. Datasets containing artworks