Towards Integration of Embodiment Features for Prosodic Prominence Prediction from Text

Abstract

Prosodic prominence prediction is an important task in the area of speech processing and especially forms an essential part of modern text-to-speech systems. Previous work has broadly focused on acoustic and linguistic features (such as syntactic and semantic features) for predicting prosodic prominence. However, human models of prosody are known to be highly multimodal and grounded on denotations of physical entities and embodied experience. In this paper we present a first study where we integrate multimodal sensorimotor associations by exploiting the Lancaster Sensorimotor Norms towards prosodic prominence prediction. Our results highlight the importance of sensorimotor knowledge especially for models in low-data regimens where we show that it improves the performance by a significant margin.