Sean Kates, Tine Paulsen, Sidak Yntiso, Joshua A. Tucker
Abstract Many large survey courses rely on multiple professors or teaching assistants to judge student responses to open-ended questions. Even following best practices, students with similar levels of conceptual understanding can receive widely varying assessments from different graders. We detail how this can occur and argue that it is an example of differential item functioning (or interpersonal incomparability), where graders interpret the same possible grading range differently. Using both actual assessment data from a large survey course in Comparative Politics and simulation methods, we show that the bias can be corrected by a small number of “bridging” observations across graders. We conclude by offering best practices for fair assessment in large survey courses.
{"title":"Bridging the Grade Gap: Reducing Assessment Bias in a Multi-Grader Class","authors":"Sean Kates, Tine Paulsen, Sidak Yntiso, Joshua A. Tucker","doi":"10.1017/pan.2022.27","DOIUrl":"https://doi.org/10.1017/pan.2022.27","url":null,"abstract":"Abstract Many large survey courses rely on multiple professors or teaching assistants to judge student responses to open-ended questions. Even following best practices, students with similar levels of conceptual understanding can receive widely varying assessments from different graders. We detail how this can occur and argue that it is an example of differential item functioning (or interpersonal incomparability), where graders interpret the same possible grading range differently. Using both actual assessment data from a large survey course in Comparative Politics and simulation methods, we show that the bias can be corrected by a small number of “bridging” observations across graders. We conclude by offering best practices for fair assessment in large survey courses.","PeriodicalId":48270,"journal":{"name":"Political Analysis","volume":"31 1","pages":"642 - 650"},"PeriodicalIF":5.4,"publicationDate":"2022-12-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47404335","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract Video advertisements, either through television or the Internet, play an essential role in modern political campaigns. For over two decades, researchers have studied television video ads by analyzing the hand-coded data from the Wisconsin Advertising Project and its successor, the Wesleyan Media Project (WMP). Unfortunately, manually coding more than a hundred of variables, such as issue mentions, opponent appearance, and negativity, for many videos is a laborious and expensive process. We propose to automatically code campaign advertisement videos. Applying state-of-the-art machine learning methods, we extract various audio and image features from each video file. We show that our machine coding is comparable to human coding for many variables of the WMP datasets. Since many candidates make their advertisement videos available on the Internet, automated coding can dramatically improve the efficiency and scope of campaign advertisement research. Open-source software package is available for implementing the proposed methodology.
{"title":"Automated Coding of Political Campaign Advertisement Videos: An Empirical Validation Study","authors":"Alexander Tarr, June Hwang, Kosuke Imai","doi":"10.1017/pan.2022.26","DOIUrl":"https://doi.org/10.1017/pan.2022.26","url":null,"abstract":"Abstract Video advertisements, either through television or the Internet, play an essential role in modern political campaigns. For over two decades, researchers have studied television video ads by analyzing the hand-coded data from the Wisconsin Advertising Project and its successor, the Wesleyan Media Project (WMP). Unfortunately, manually coding more than a hundred of variables, such as issue mentions, opponent appearance, and negativity, for many videos is a laborious and expensive process. We propose to automatically code campaign advertisement videos. Applying state-of-the-art machine learning methods, we extract various audio and image features from each video file. We show that our machine coding is comparable to human coding for many variables of the WMP datasets. Since many candidates make their advertisement videos available on the Internet, automated coding can dramatically improve the efficiency and scope of campaign advertisement research. Open-source software package is available for implementing the proposed methodology.","PeriodicalId":48270,"journal":{"name":"Political Analysis","volume":"31 1","pages":"554 - 574"},"PeriodicalIF":5.4,"publicationDate":"2022-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43838742","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract Recent research has shown that interaction effects may often be nonlinear (Hainmueller, Mummolo, and Xu [2019, Political Analysis 27, 163–192]). As standard interaction effect specifications assume a linear interaction effect, that is, the moderator conditions the effect at a constant rate, this can lead to bias. However, allowing nonlinear interaction effects, without accounting for other nonlinearities and nonlinear interaction effects, can also lead to biased estimates. Specifically, researchers can infer nonlinear interaction effects, even though the true interaction effect is linear, when variables used for covariate adjustment that are correlated with the moderator have a nonlinear effect upon the outcome of interest. We illustrate this bias with simulations and show how diagnostic tools recommended in the literature are unable to uncover the issue. We show how using the adaptive Lasso to identify relevant nonlinearities among variables used for covariate adjustment can avoid this issue. Moreover, the use of regularized estimators, which allow for a fuller set of nonlinearities, both independent and interactive, is more generally shown to avoid this bias and more general forms of omitted interaction bias.
最近的研究表明，相互作用效应往往是非线性的(Hainmueller, Mummolo, and Xu [2019, Political Analysis 27, 163-192])。由于标准的相互作用效应规范假设了线性的相互作用效应，即慢化剂以恒定的速率调节作用，这可能导致偏差。然而，允许非线性相互作用效应，而不考虑其他非线性和非线性相互作用效应，也可能导致有偏差的估计。具体来说，研究人员可以推断非线性相互作用效应，即使真正的相互作用效应是线性的，当用于协变量调整的变量与调节因子相关时，对感兴趣的结果产生非线性影响。我们通过模拟来说明这种偏差，并展示了文献中推荐的诊断工具如何无法发现问题。我们展示了如何使用自适应Lasso来识别用于协变量调整的变量之间的相关非线性可以避免这个问题。此外，正则估计的使用，它允许更全面的非线性集，既独立又相互作用，更普遍地显示避免这种偏差和更一般形式的忽略的相互作用偏差。
{"title":"The Consequences of Model Misspecification for the Estimation of Nonlinear Interaction Effects","authors":"Janina Beiser-McGrath, Liam F. Beiser-McGrath","doi":"10.1017/pan.2022.25","DOIUrl":"https://doi.org/10.1017/pan.2022.25","url":null,"abstract":"Abstract Recent research has shown that interaction effects may often be nonlinear (Hainmueller, Mummolo, and Xu [2019, Political Analysis 27, 163–192]). As standard interaction effect specifications assume a linear interaction effect, that is, the moderator conditions the effect at a constant rate, this can lead to bias. However, allowing nonlinear interaction effects, without accounting for other nonlinearities and nonlinear interaction effects, can also lead to biased estimates. Specifically, researchers can infer nonlinear interaction effects, even though the true interaction effect is linear, when variables used for covariate adjustment that are correlated with the moderator have a nonlinear effect upon the outcome of interest. We illustrate this bias with simulations and show how diagnostic tools recommended in the literature are unable to uncover the issue. We show how using the adaptive Lasso to identify relevant nonlinearities among variables used for covariate adjustment can avoid this issue. Moreover, the use of regularized estimators, which allow for a fuller set of nonlinearities, both independent and interactive, is more generally shown to avoid this bias and more general forms of omitted interaction bias.","PeriodicalId":48270,"journal":{"name":"Political Analysis","volume":"31 1","pages":"278 - 287"},"PeriodicalIF":5.4,"publicationDate":"2022-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45197888","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract Theory has long suggested that swing voting is a response to cross-pressures arising from a mix of individual attributes and contextual factors. Unfortunately, existing regression-based approaches are ill-suited to explore the complex combinations of demographic, policy, and political factors that produce swing voters in American elections. This gap between theory and practice motivates our use of an ensemble of supervised machine learning methods to predict swing voters in the 2012, 2016, and 2020 U.S. presidential elections. The results from the learning ensemble substantiate the existence of swing voters in contemporary American elections. Specifically, we demonstrate that the learning ensemble produces well-calibrated and externally valid predictions of swing voter propensity in later elections and for related behaviors such as split-ticket voting. Although interpreting black-box models is more challenging, they can nonetheless provide meaningful substantive insights meriting further exploration. Here, we use flexible model-agnostic tools to perturb the ensemble and demonstrate that cross-pressures (particularly those involving ideological and policy-related considerations) are essential to accurately predict swing voters.
{"title":"Measuring Swing Voters with a Supervised Machine Learning Ensemble","authors":"Christopher Hare, Mikayla Kutsuris","doi":"10.1017/pan.2022.24","DOIUrl":"https://doi.org/10.1017/pan.2022.24","url":null,"abstract":"Abstract Theory has long suggested that swing voting is a response to cross-pressures arising from a mix of individual attributes and contextual factors. Unfortunately, existing regression-based approaches are ill-suited to explore the complex combinations of demographic, policy, and political factors that produce swing voters in American elections. This gap between theory and practice motivates our use of an ensemble of supervised machine learning methods to predict swing voters in the 2012, 2016, and 2020 U.S. presidential elections. The results from the learning ensemble substantiate the existence of swing voters in contemporary American elections. Specifically, we demonstrate that the learning ensemble produces well-calibrated and externally valid predictions of swing voter propensity in later elections and for related behaviors such as split-ticket voting. Although interpreting black-box models is more challenging, they can nonetheless provide meaningful substantive insights meriting further exploration. Here, we use flexible model-agnostic tools to perturb the ensemble and demonstrate that cross-pressures (particularly those involving ideological and policy-related considerations) are essential to accurately predict swing voters.","PeriodicalId":48270,"journal":{"name":"Political Analysis","volume":"31 1","pages":"537 - 553"},"PeriodicalIF":5.4,"publicationDate":"2022-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46221099","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract Researchers of time series cross-sectional data regularly face the change-point problem, which requires them to discern between significant parametric shifts that can be deemed structural changes and minor parametric shifts that must be considered noise. In this paper, we develop a general Bayesian method for change-point detection in high-dimensional data and present its application in the context of the fixed-effect model. Our proposed method, hidden Markov Bayesian bridge model, jointly estimates high-dimensional regime-specific parameters and hidden regime transitions in a unified way. We apply our method to Alvarez, Garrett, and Lange’s (1991, American Political Science Review 85, 539–556) study of the relationship between government partisanship and economic growth and Allee and Scalera’s (2012, International Organization 66, 243–276) study of membership effects in international organizations. In both applications, we found that the proposed method successfully identify substantively meaningful temporal heterogeneity in parameters of regression models.
时间序列横截面数据的研究人员经常面临变化点问题，这需要他们区分可视为结构变化的显著参数变化和必须视为噪声的微小参数变化。本文提出了一种用于高维数据变点检测的通用贝叶斯方法，并给出了该方法在固定效应模型中的应用。我们提出的隐马尔可夫贝叶斯桥模型以统一的方式联合估计高维状态特定参数和隐藏状态转移。我们将我们的方法应用于Alvarez、Garrett和Lange (1991, American Political Science Review 85, 539-556)关于政府党派关系与经济增长关系的研究，以及Allee和Scalera (2012, International Organization 66, 243-276)关于国际组织成员效应的研究。在这两个应用中，我们发现所提出的方法成功地识别了回归模型参数中具有实质性意义的时间异质性。
{"title":"Change-Point Detection and Regularization in Time Series Cross-Sectional Data Analysis","authors":"J. Park, S. Yamauchi","doi":"10.1017/pan.2022.23","DOIUrl":"https://doi.org/10.1017/pan.2022.23","url":null,"abstract":"Abstract Researchers of time series cross-sectional data regularly face the change-point problem, which requires them to discern between significant parametric shifts that can be deemed structural changes and minor parametric shifts that must be considered noise. In this paper, we develop a general Bayesian method for change-point detection in high-dimensional data and present its application in the context of the fixed-effect model. Our proposed method, hidden Markov Bayesian bridge model, jointly estimates high-dimensional regime-specific parameters and hidden regime transitions in a unified way. We apply our method to Alvarez, Garrett, and Lange’s (1991, American Political Science Review 85, 539–556) study of the relationship between government partisanship and economic growth and Allee and Scalera’s (2012, International Organization 66, 243–276) study of membership effects in international organizations. In both applications, we found that the proposed method successfully identify substantively meaningful temporal heterogeneity in parameters of regression models.","PeriodicalId":48270,"journal":{"name":"Political Analysis","volume":"31 1","pages":"257 - 277"},"PeriodicalIF":5.4,"publicationDate":"2022-10-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43366040","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract I propose a new model, ordered Beta regression, for continuous distributions with both lower and upper bounds, such as data arising from survey slider scales, visual analog scales, and dose–response relationships. This model employs the cut point technique popularized by ordered logit to fit a single linear model to both continuous (0,1) and degenerate [0,1] responses. The model can be estimated with or without observations at the bounds, and as such is a general solution for these types of data. Employing a Monte Carlo simulation, I show that the model is noticeably more efficient than ordinary least squares regression, zero-and-one-inflated Beta regression, rescaled Beta regression, and fractional logit while fully capturing nuances in the outcome. I apply the model to a replication of the Aidt and Jensen (2014, European Economic Review 72, 52–75) study of suffrage extensions in Europe. The model can be fit with the R package ordbetareg to facilitate hierarchical, dynamic, and multivariate modeling.
{"title":"Ordered Beta Regression: A Parsimonious, Well-Fitting Model for Continuous Data with Lower and Upper Bounds","authors":"R. Kubinec","doi":"10.1017/pan.2022.20","DOIUrl":"https://doi.org/10.1017/pan.2022.20","url":null,"abstract":"Abstract I propose a new model, ordered Beta regression, for continuous distributions with both lower and upper bounds, such as data arising from survey slider scales, visual analog scales, and dose–response relationships. This model employs the cut point technique popularized by ordered logit to fit a single linear model to both continuous (0,1) and degenerate [0,1] responses. The model can be estimated with or without observations at the bounds, and as such is a general solution for these types of data. Employing a Monte Carlo simulation, I show that the model is noticeably more efficient than ordinary least squares regression, zero-and-one-inflated Beta regression, rescaled Beta regression, and fractional logit while fully capturing nuances in the outcome. I apply the model to a replication of the Aidt and Jensen (2014, European Economic Review 72, 52–75) study of suffrage extensions in Europe. The model can be fit with the R package ordbetareg to facilitate hierarchical, dynamic, and multivariate modeling.","PeriodicalId":48270,"journal":{"name":"Political Analysis","volume":"31 1","pages":"519 - 536"},"PeriodicalIF":5.4,"publicationDate":"2022-07-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42600884","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Kirk Bansak, Jens Hainmueller, D. Hopkins, Teppei Yamamoto
Abstract Political scientists have increasingly deployed conjoint survey experiments to understand multidimensional choices in various settings. In this paper, we show that the average marginal component effect (AMCE) constitutes an aggregation of individual-level preferences that is meaningful both theoretically and empirically. First, extending previous results to allow for arbitrary randomization distributions, we show how the AMCE represents a summary of voters’ multidimensional preferences that combines directionality and intensity according to a probabilistic generalization of the Borda rule. We demonstrate why incorporating both the directionality and intensity of multi-attribute preferences is essential for analyzing real-world elections, in which ceteris paribus comparisons almost never occur. Second, and in further empirical support of this point, we show how this aggregation translates directly into a primary quantity of interest to election scholars: the effect of a change in an attribute on a candidate’s or party’s expected vote share. These properties hold irrespective of the heterogeneity, strength, or interactivity of voters’ preferences and regardless of how votes are aggregated into seats. Finally, we propose, formalize, and evaluate the feasibility of using conjoint data to estimate alternative quantities of interest to electoral studies, including the effect of an attribute on the probability of winning.
{"title":"Using Conjoint Experiments to Analyze Election Outcomes: The Essential Role of the Average Marginal Component Effect","authors":"Kirk Bansak, Jens Hainmueller, D. Hopkins, Teppei Yamamoto","doi":"10.1017/pan.2022.16","DOIUrl":"https://doi.org/10.1017/pan.2022.16","url":null,"abstract":"Abstract Political scientists have increasingly deployed conjoint survey experiments to understand multidimensional choices in various settings. In this paper, we show that the average marginal component effect (AMCE) constitutes an aggregation of individual-level preferences that is meaningful both theoretically and empirically. First, extending previous results to allow for arbitrary randomization distributions, we show how the AMCE represents a summary of voters’ multidimensional preferences that combines directionality and intensity according to a probabilistic generalization of the Borda rule. We demonstrate why incorporating both the directionality and intensity of multi-attribute preferences is essential for analyzing real-world elections, in which ceteris paribus comparisons almost never occur. Second, and in further empirical support of this point, we show how this aggregation translates directly into a primary quantity of interest to election scholars: the effect of a change in an attribute on a candidate’s or party’s expected vote share. These properties hold irrespective of the heterogeneity, strength, or interactivity of voters’ preferences and regardless of how votes are aggregated into seats. Finally, we propose, formalize, and evaluate the feasibility of using conjoint data to estimate alternative quantities of interest to electoral studies, including the effect of an attribute on the probability of winning.","PeriodicalId":48270,"journal":{"name":"Political Analysis","volume":"31 1","pages":"500 - 518"},"PeriodicalIF":5.4,"publicationDate":"2022-06-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44645179","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：