Investigating Opinions on Public Policies in Digital Media: Setting up a Supervised Machine Learning Tool for Stance Classification

IF 6.3 1区 文学 Q1 COMMUNICATION Communication Methods and Measures Pub Date : 2022-12-12 DOI:10.1080/19312458.2022.2151579
Christina Viehmann, Tilman Beck, Marcus Maurer, Oliver Quiring, Iryna Gurevych
{"title":"Investigating Opinions on Public Policies in Digital Media: Setting up a Supervised Machine Learning Tool for Stance Classification","authors":"Christina Viehmann, Tilman Beck, Marcus Maurer, Oliver Quiring, Iryna Gurevych","doi":"10.1080/19312458.2022.2151579","DOIUrl":null,"url":null,"abstract":"ABSTRACT Supervised machine learning (SML) provides us with tools to efficiently scrutinize large corpora of communication texts. Yet, setting up such a tool involves plenty of decisions starting with the data needed for training, the selection of an algorithm, and the details of model training. We aim at establishing a firm link between communication research tasks and the corresponding state-of-the-art in natural language processing research by systematically comparing the performance of different automatic text analysis approaches. We do this for a challenging task – stance detection of opinions on policy measures to tackle the COVID-19 pandemic in Germany voiced on Twitter. Our results add evidence that pre-trained language models such as BERT outperform feature-based and other neural network approaches. Yet, the gains one can achieve differ greatly depending on the specific merits of pre-training (i.e., use of different language models). Adding to the robustness of our conclusions, we run a generalizability check with a different use case in terms of language and topic. Additionally, we illustrate how the amount and quality of training data affect model performance pointing to potential compensation effects. Based on our results, we derive important practical recommendations for setting up such SML tools to study communication texts.","PeriodicalId":47552,"journal":{"name":"Communication Methods and Measures","volume":null,"pages":null},"PeriodicalIF":6.3000,"publicationDate":"2022-12-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"2","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Communication Methods and Measures","FirstCategoryId":"98","ListUrlMain":"https://doi.org/10.1080/19312458.2022.2151579","RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMMUNICATION","Score":null,"Total":0}
引用次数: 2

Abstract

ABSTRACT Supervised machine learning (SML) provides us with tools to efficiently scrutinize large corpora of communication texts. Yet, setting up such a tool involves plenty of decisions starting with the data needed for training, the selection of an algorithm, and the details of model training. We aim at establishing a firm link between communication research tasks and the corresponding state-of-the-art in natural language processing research by systematically comparing the performance of different automatic text analysis approaches. We do this for a challenging task – stance detection of opinions on policy measures to tackle the COVID-19 pandemic in Germany voiced on Twitter. Our results add evidence that pre-trained language models such as BERT outperform feature-based and other neural network approaches. Yet, the gains one can achieve differ greatly depending on the specific merits of pre-training (i.e., use of different language models). Adding to the robustness of our conclusions, we run a generalizability check with a different use case in terms of language and topic. Additionally, we illustrate how the amount and quality of training data affect model performance pointing to potential compensation effects. Based on our results, we derive important practical recommendations for setting up such SML tools to study communication texts.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
调查数字媒体公共政策的意见:建立一个有监督的立场分类机器学习工具
监督式机器学习(SML)为我们提供了有效检查大型交流文本语料库的工具。然而,建立这样一个工具涉及大量的决策,从训练所需的数据、算法的选择和模型训练的细节开始。我们的目标是通过系统地比较不同自动文本分析方法的性能,在通信研究任务和自然语言处理研究的相应技术之间建立牢固的联系。我们这样做是为了一项具有挑战性的任务,即对推特上针对德国新冠肺炎疫情的政策措施的意见进行立场检测。我们的研究结果进一步证明,预训练语言模型(如BERT)优于基于特征的方法和其他神经网络方法。然而,根据预训练的具体优点(即使用不同的语言模型),可以获得的收益差异很大。为了增加结论的健壮性,我们根据语言和主题对不同的用例进行了通用性检查。此外,我们说明了训练数据的数量和质量如何影响模型性能,指出潜在的补偿效应。基于我们的结果,我们得出了一些重要的实用建议,用于设置这样的SML工具来研究交流文本。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
CiteScore
21.10
自引率
1.80%
发文量
9
期刊介绍: Communication Methods and Measures aims to achieve several goals in the field of communication research. Firstly, it aims to bring attention to and showcase developments in both qualitative and quantitative research methodologies to communication scholars. This journal serves as a platform for researchers across the field to discuss and disseminate methodological tools and approaches. Additionally, Communication Methods and Measures seeks to improve research design and analysis practices by offering suggestions for improvement. It aims to introduce new methods of measurement that are valuable to communication scientists or enhance existing methods. The journal encourages submissions that focus on methods for enhancing research design and theory testing, employing both quantitative and qualitative approaches. Furthermore, the journal is open to articles devoted to exploring the epistemological aspects relevant to communication research methodologies. It welcomes well-written manuscripts that demonstrate the use of methods and articles that highlight the advantages of lesser-known or newer methods over those traditionally used in communication. In summary, Communication Methods and Measures strives to advance the field of communication research by showcasing and discussing innovative methodologies, improving research practices, and introducing new measurement methods.
期刊最新文献
JST and rJST: joint estimation of sentiment and topics in textual data using a semi-supervised approach Using State Space Grids to Quantify and Examine Dynamics of Dyadic Conversation Bootstrapping public entities. Domain-specific NER for public speakers On Measurement Validity and Language Models: Increasing Validity and Decreasing Bias with Instructions Googling Politics? Comparing Five Computational Methods to Identify Political and News-related Searches from Web Browser Histories
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1