Contextualized vs. Static Word Embeddings for Word-based Analysis of Opposing Opinions

Abstract

Word embeddings are useful for studying public opinions by summarizing opinions about a concept by finding the nearest neighbors in the word embedding space. Static word embeddings such as word2vec are powerful for handling large amounts of text, while contextualized word embeddings from transformer-based models yield better embeddings by some evaluation metrics. In this study, we explore the differences between static and contextualized embeddings for word-based analysis of opposing opinions. We find that pre-training is necessary for static embeddings when the corpus is small, but contextualized embeddings are superior. When the focus corpus is large, static embeddings reflect related concepts, while contextualized embeddings often show synonyms or cohypernyms. Static embeddings trained only on the focus corpus capture opposing opinions better than contextualized embeddings.