Conceptual Combination in Large Language Models: Uncovering Implicit Relational Interpretations in Compound Words With Contextualized Word Embeddings

Abstract

Large language models (LLMs) have been proposed as candidate models of human semantics, and as such, they must be able to account for conceptual combination. This work explores the ability of two LLMs, namely, BERT-base and Llama-2-13b, to reveal the implicit meaning of existing and novel compound words. According to psycholinguistic theories, understanding the meaning of a compound (e.g., “snowman”) involves its automatic decomposition into constituent meanings (“snow,” “man”), which are then connected by an implicit semantic relation selected from a set of possible competitors (FOR, MADE OF, BY, …) to obtain a plausible interpretation (“man MADE OF snow”). Here, we leverage the flexibility of LLMs to obtain contextualized representations for both target compounds (e.g., “snowman”) and their implicit interpretations (e.g., “man MADE OF snow”). We demonstrate that replacing a compound with a paraphrased version leads to changes to the embeddings that are inversely proportional to the paraphrase's plausibility, estimated by human raters. While this relation holds for both existing and novel compounds, results obtained for novel compounds are substantially weaker, and older distributional models outperform LLMs. Nonetheless, the present results show that LLMs can offer a valid approximation of the internal structure of compound words posited by cognitive theories, thus representing a promising tool to model word senses that are at once implicit and possible.