Pub Date : 2024-07-25DOI: 10.1177/10944281241261913
James A. Grand, Michael T. Braun, Goran Kuljanin
Computational modeling holds significant promise as a tool for improving how theory is developed, expressed, and used to inform empirical research and evaluation efforts. However, the knowledge and skillsets needed to build computational models are rarely developed in the training received by social and organizational scientists. The purpose of this manuscript is to provide an accessible introduction to and reference for building computational models to represent theory. We first discuss important principles and recommendations for “thinking about” theory and developing explanatory accounts in ways that facilitate translating their core assumptions, specifications, and ideas into a computational model. Next, we address some frequently asked questions related to building computational models that introduce several fundamental tasks/concepts involved in building models to represent theory and demonstrate how they can be implemented in the R programming language to produce executable model code. The accompanying supplemental materials describes additional considerations relevant to building and using computational models, provides multiple examples of complete computational model code written in R, and an interactive application offering guided practice on key model-building tasks/concepts in R.
计算模型作为一种工具，在改进理论的开发、表达和使用方式，为实证研究和评估工作提供信息方面大有可为。然而，建立计算模型所需的知识和技能很少在社会和组织科学家接受的培训中得到发展。本手稿的目的是为建立代表理论的计算模型提供通俗易懂的介绍和参考。我们首先讨论了 "思考 "理论和开发解释性描述的重要原则和建议，这些原则和建议有助于将理论的核心假设、规范和观点转化为计算模型。接下来，我们讨论了一些与建立计算模型有关的常见问题，介绍了建立模型以表示理论所涉及的几项基本任务/概念，并演示了如何用 R 编程语言实现这些任务/概念，以生成可执行的模型代码。随书附赠的补充材料介绍了与构建和使用计算模型相关的其他注意事项，提供了多个用 R 语言编写的完整计算模型代码示例，并提供了一个交互式应用程序，指导读者练习用 R 语言构建模型的关键任务/概念。
{"title":"Hello World! Building Computational Models to Represent Social and Organizational Theory","authors":"James A. Grand, Michael T. Braun, Goran Kuljanin","doi":"10.1177/10944281241261913","DOIUrl":"https://doi.org/10.1177/10944281241261913","url":null,"abstract":"Computational modeling holds significant promise as a tool for improving how theory is developed, expressed, and used to inform empirical research and evaluation efforts. However, the knowledge and skillsets needed to build computational models are rarely developed in the training received by social and organizational scientists. The purpose of this manuscript is to provide an accessible introduction to and reference for building computational models to represent theory. We first discuss important principles and recommendations for “thinking about” theory and developing explanatory accounts in ways that facilitate translating their core assumptions, specifications, and ideas into a computational model. Next, we address some frequently asked questions related to building computational models that introduce several fundamental tasks/concepts involved in building models to represent theory and demonstrate how they can be implemented in the R programming language to produce executable model code. The accompanying supplemental materials describes additional considerations relevant to building and using computational models, provides multiple examples of complete computational model code written in R, and an interactive application offering guided practice on key model-building tasks/concepts in R.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"1 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2024-07-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141764124","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-07-25DOI: 10.1177/10944281241264027
Louis Hickman, Josh Liff, Caleb Rottman, Charles Calderwood
While machine learning (ML) can validly score psychological constructs from behavior, several conditions often change across studies, making it difficult to understand why the psychometric properties of ML models differ across studies. We address this gap in the context of automatically scored interviews. Across multiple datasets, for interview- or question-level scoring of self-reported, tested, and interviewer-rated constructs, we manipulate the training sample size and natural language processing (NLP) method while observing differences in ground truth reliability. We examine how these factors influence the ML model scores’ test–retest reliability and convergence, and we develop multilevel models for estimating the convergent-related validity of ML model scores in similar interviews. When the ground truth is interviewer ratings, hundreds of observations are adequate for research purposes, while larger samples are recommended for practitioners to support generalizability across populations and time. However, self-reports and tested constructs require larger training samples. Particularly when the ground truth is interviewer ratings, NLP embedding methods improve upon count-based methods. Given mixed findings regarding ground truth reliability, we discuss future research possibilities on factors that affect supervised ML models’ psychometric properties.
虽然机器学习（ML）可以有效地从行为中对心理结构进行评分，但在不同的研究中，有几个条件经常会发生变化，因此很难理解为什么不同研究中的 ML 模型的心理测量特性会有所不同。我们在自动评分访谈中解决了这一空白。在多个数据集中，对于自我报告、测试和面试官评分的访谈或问题级评分，我们操纵了训练样本大小和自然语言处理（NLP）方法，同时观察了基本真实可靠性的差异。我们研究了这些因素如何影响 ML 模型得分的重测可靠性和收敛性，并开发了多层次模型来估计类似访谈中 ML 模型得分的收敛性相关有效性。当基本事实是访谈者的评分时，数百个观察样本就足以满足研究目的，而对于从业人员来说，则建议使用更大的样本，以支持跨人群和跨时间的普适性。然而，自我报告和经过测试的结构需要更大的训练样本。特别是当基本真实情况是访谈者的评分时，NLP 嵌入方法比基于计数的方法更有优势。鉴于有关基本真实可靠性的研究结果好坏参半，我们讨论了未来研究影响有监督 ML 模型心理计量特性的因素的可能性。
{"title":"The Effects of the Training Sample Size, Ground Truth Reliability, and NLP Method on Language-Based Automatic Interview Scores’ Psychometric Properties","authors":"Louis Hickman, Josh Liff, Caleb Rottman, Charles Calderwood","doi":"10.1177/10944281241264027","DOIUrl":"https://doi.org/10.1177/10944281241264027","url":null,"abstract":"While machine learning (ML) can validly score psychological constructs from behavior, several conditions often change across studies, making it difficult to understand why the psychometric properties of ML models differ across studies. We address this gap in the context of automatically scored interviews. Across multiple datasets, for interview- or question-level scoring of self-reported, tested, and interviewer-rated constructs, we manipulate the training sample size and natural language processing (NLP) method while observing differences in ground truth reliability. We examine how these factors influence the ML model scores’ test–retest reliability and convergence, and we develop multilevel models for estimating the convergent-related validity of ML model scores in similar interviews. When the ground truth is interviewer ratings, hundreds of observations are adequate for research purposes, while larger samples are recommended for practitioners to support generalizability across populations and time. However, self-reports and tested constructs require larger training samples. Particularly when the ground truth is interviewer ratings, NLP embedding methods improve upon count-based methods. Given mixed findings regarding ground truth reliability, we discuss future research possibilities on factors that affect supervised ML models’ psychometric properties.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"37 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2024-07-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141764243","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-18eCollection Date: 2025-07-01DOI: 10.1177/10944281241259075
Christian Rupietta, Johannes Meuer
In the past 20 years, researchers have significantly advanced various management fields by examining organizational phenomena through a configurational lens, including competitive strategies, corporate governance mechanisms, and innovation systems. Qualitative comparative analysis (QCA) has emerged as a primary method for empirically investigating organizational configurations. However, QCA has traditionally struggled to capture the temporal aspects of configurational phenomena. In this paper, we present configurational comparative process analysis (C2PA), which merges QCA with sequence analysis. We introduce the concept of configurational themes-recognizable temporal patterns of recurring combinations of explanatory conditions-to identify and track the temporal dynamics among these phenomena. We also outline configurational matching-a method for empirically identifying these themes by distinguishing theme-defining from theme-supporting conditions. C2PA allows researchers to explore the temporal dynamics of configurational phenomena, such as their stability, emergence, and decline at critical junctures. We illustrate the application of C2PA through a study of shareholder value orientation and discuss its potential for addressing key questions in management research.
{"title":"Comparative Configurational Process Analysis: A New Set-Theoretic Technique for Longitudinal Case Analysis.","authors":"Christian Rupietta, Johannes Meuer","doi":"10.1177/10944281241259075","DOIUrl":"10.1177/10944281241259075","url":null,"abstract":"<p><p>In the past 20 years, researchers have significantly advanced various management fields by examining organizational phenomena through a configurational lens, including competitive strategies, corporate governance mechanisms, and innovation systems. Qualitative comparative analysis (QCA) has emerged as a primary method for empirically investigating organizational configurations. However, QCA has traditionally struggled to capture the temporal aspects of configurational phenomena. In this paper, we present configurational comparative process analysis (C<sup>2</sup>PA), which merges QCA with sequence analysis. We introduce the concept of configurational themes-recognizable temporal patterns of recurring combinations of explanatory conditions-to identify and track the temporal dynamics among these phenomena. We also outline configurational matching-a method for empirically identifying these themes by distinguishing theme-defining from theme-supporting conditions. C<sup>2</sup>PA allows researchers to explore the temporal dynamics of configurational phenomena, such as their stability, emergence, and decline at critical junctures. We illustrate the application of C<sup>2</sup>PA through a study of shareholder value orientation and discuss its potential for addressing key questions in management research.</p>","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"28 3","pages":"405-432"},"PeriodicalIF":8.9,"publicationDate":"2024-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12225975/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144576001","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-05-02DOI: 10.1177/10944281241246772
Wen Wei Loh, Dongning Ren
Understanding the experiences of vulnerable workers is an important scientific pursuit. For example, research interest is often in quantifying the impacts of adverse exposures such as discrimination, exclusion, harassment, or job insecurity, among others. However, routine approaches have only focused on the average treatment effect, which encapsulates the impact of an exposure (e.g., discrimination) applied to the entire study population—including those who were not exposed. In this paper, we propose using a more refined causal quantity uniquely suited to address such causal queries: The effect of treatment on the treated (ETT) from the causal inference literature. We explain why the ETT is a more pertinent causal estimand for investigating the experiences of vulnerable workers by highlighting three appealing features: Better interpretability, greater accuracy, and enhanced robustness to violations of empirically untestable causal assumptions. We further describe how to estimate the ETT by introducing and comparing two estimators. Both estimators are conferred with a so-called doubly robust property. We hope the current proposal empowers organizational scholars in their crucial endeavors dedicated to understanding the vulnerable workforce.
了解弱势工人的经历是一项重要的科学追求。例如，研究兴趣往往在于量化歧视、排斥、骚扰或工作不稳定等不利暴露的影响。然而，常规方法只关注平均处理效果，即某一暴露（如歧视）对整个研究人群（包括未暴露人群）的影响。在本文中，我们建议使用一种更精细的因果量，它非常适合解决此类因果问题：因果推断文献中的治疗对被治疗者的影响（ETT）。我们通过强调三个吸引人的特点来解释为什么 ETT 是调查弱势工人经历的更相关的因果估计量：更好的可解释性、更高的准确性以及对违反经验上无法检验的因果假设的稳健性。通过介绍和比较两种估计方法，我们进一步介绍了如何估计 ETT。这两个估计器都具有所谓的双重稳健性。我们希望当前的建议能够增强组织学者的能力，使他们能够致力于了解弱势劳动力的重要工作。
{"title":"Enhancing Causal Pursuits in Organizational Science: Targeting the Effect of Treatment on the Treated in Research on Vulnerable Populations","authors":"Wen Wei Loh, Dongning Ren","doi":"10.1177/10944281241246772","DOIUrl":"https://doi.org/10.1177/10944281241246772","url":null,"abstract":"Understanding the experiences of vulnerable workers is an important scientific pursuit. For example, research interest is often in quantifying the impacts of adverse exposures such as discrimination, exclusion, harassment, or job insecurity, among others. However, routine approaches have only focused on the average treatment effect, which encapsulates the impact of an exposure (e.g., discrimination) applied to the entire study population—including those who were not exposed. In this paper, we propose using a more refined causal quantity uniquely suited to address such causal queries: The effect of treatment on the treated (ETT) from the causal inference literature. We explain why the ETT is a more pertinent causal estimand for investigating the experiences of vulnerable workers by highlighting three appealing features: Better interpretability, greater accuracy, and enhanced robustness to violations of empirically untestable causal assumptions. We further describe how to estimate the ETT by introducing and comparing two estimators. Both estimators are conferred with a so-called doubly robust property. We hope the current proposal empowers organizational scholars in their crucial endeavors dedicated to understanding the vulnerable workforce.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"51 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2024-05-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140826389","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-22DOI: 10.1177/10944281241245444
Linda Jakob Sadeh, Avital Baikovich, Tammar B. Zilber
This article proposes a framework for reflexive choice in qualitative research, centering on social interaction. Interaction, fundamental to social and organizational life, has been studied extensively. Yet, researchers can get lost in the plethora of methodological tools, hampering reflexive choice. Our proposed framework consists of four dimensions of interaction (content, communication patterns, emotions, and roles), intersecting with five levels of analysis (individual, dyadic, group, organizational, and sociocultural), as well as three overarching analytic principles (following the dynamic, consequential, and contextual nature of interaction). For each intersection between dimension and level, we specify analytical questions, empirical markers, and references to exemplary works. The framework functions both as a compass, indicating potential directions for research design and data collection methods, and as a roadmap, illuminating pathways at the analysis stage. Our contributions are twofold: First, our framework fleshes out the broad spectrum of available methods for analyzing interaction, providing pragmatic tools for the researcher to reflexively choose from. Second, we highlight the broader relevance of maps, such as our own, for enhancing reflexive methodological choices.
{"title":"Analyzing Social Interaction in Organizations: A Roadmap for Reflexive Choice","authors":"Linda Jakob Sadeh, Avital Baikovich, Tammar B. Zilber","doi":"10.1177/10944281241245444","DOIUrl":"https://doi.org/10.1177/10944281241245444","url":null,"abstract":"This article proposes a framework for reflexive choice in qualitative research, centering on social interaction. Interaction, fundamental to social and organizational life, has been studied extensively. Yet, researchers can get lost in the plethora of methodological tools, hampering reflexive choice. Our proposed framework consists of four dimensions of interaction (content, communication patterns, emotions, and roles), intersecting with five levels of analysis (individual, dyadic, group, organizational, and sociocultural), as well as three overarching analytic principles (following the dynamic, consequential, and contextual nature of interaction). For each intersection between dimension and level, we specify analytical questions, empirical markers, and references to exemplary works. The framework functions both as a compass, indicating potential directions for research design and data collection methods, and as a roadmap, illuminating pathways at the analysis stage. Our contributions are twofold: First, our framework fleshes out the broad spectrum of available methods for analyzing interaction, providing pragmatic tools for the researcher to reflexively choose from. Second, we highlight the broader relevance of maps, such as our own, for enhancing reflexive methodological choices.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"9 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2024-04-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140637754","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-17DOI: 10.1177/10944281241240180
Stefanie Habersang, Markus Reihlen
Qualitative meta-studies (QMS) have emerged as a promising methodology for synthesizing qualitative research within organization and management studies. However, despite considerable progress, increasingly fragmented applications of QMS impede the advancement of the methodology. To address this issue, we review and analyze the expanding body of QMS in organization and management studies. We propose a framework that encompasses the core decisions and methodological choices in the formal QMS protocol as well as the reflective—yet often implicit—meta-practices essential for deriving meaningful results from QMS. Based on our analysis, we develop two guidelines to help researchers reflectively align formal methodological choices with the intended purpose of the QMS, which can be either confirmatory or exploratory.
{"title":"Advancing Qualitative Meta-Studies (QMS): Current Practices and Reflective Guidelines for Synthesizing Qualitative Research","authors":"Stefanie Habersang, Markus Reihlen","doi":"10.1177/10944281241240180","DOIUrl":"https://doi.org/10.1177/10944281241240180","url":null,"abstract":"Qualitative meta-studies (QMS) have emerged as a promising methodology for synthesizing qualitative research within organization and management studies. However, despite considerable progress, increasingly fragmented applications of QMS impede the advancement of the methodology. To address this issue, we review and analyze the expanding body of QMS in organization and management studies. We propose a framework that encompasses the core decisions and methodological choices in the formal QMS protocol as well as the reflective—yet often implicit—meta-practices essential for deriving meaningful results from QMS. Based on our analysis, we develop two guidelines to help researchers reflectively align formal methodological choices with the intended purpose of the QMS, which can be either confirmatory or exploratory.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"25 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2024-04-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140608152","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-17DOI: 10.1177/10944281241246770
Anand P. A. van Zelderen, Theodore C. Masters-Waage, Nicky Dries, Jochen I. Menges, Diana R. Sanchez
Due to recent technological developments, vignette studies that have traditionally been done in text or video formats can now be done in immersive formats using virtual reality—but are such virtual reality video vignettes superior to traditional vignettes? To address this question, we examine participants’ experiences within a fictitious organization by comparing their responses to a relevant and particularly sensitive organizational phenomenon presented either through written text, a video recording, or a virtual reality experience. The results indicate that participants prefer more immersive methods, and that these increase their attention to critical study details. Moreover, this augments the effect sizes of several measured employee reactions—particularly those with high emotional content—suggesting that virtual reality technology offers a promising avenue for developing ecologically valid vignette studies to measure employee affect. To facilitate and expediate the use of virtual reality video vignettes in organizational research, we provide organizational scholars with a step-by-step instructional guide to develop immersive vignette studies.
{"title":"Simulating Virtual Organizations for Research: A Comparative Empirical Evaluation of Text-Based, Video, and Virtual Reality Video Vignettes","authors":"Anand P. A. van Zelderen, Theodore C. Masters-Waage, Nicky Dries, Jochen I. Menges, Diana R. Sanchez","doi":"10.1177/10944281241246770","DOIUrl":"https://doi.org/10.1177/10944281241246770","url":null,"abstract":"Due to recent technological developments, vignette studies that have traditionally been done in text or video formats can now be done in immersive formats using virtual reality—but are such virtual reality video vignettes superior to traditional vignettes? To address this question, we examine participants’ experiences within a fictitious organization by comparing their responses to a relevant and particularly sensitive organizational phenomenon presented either through written text, a video recording, or a virtual reality experience. The results indicate that participants prefer more immersive methods, and that these increase their attention to critical study details. Moreover, this augments the effect sizes of several measured employee reactions—particularly those with high emotional content—suggesting that virtual reality technology offers a promising avenue for developing ecologically valid vignette studies to measure employee affect. To facilitate and expediate the use of virtual reality video vignettes in organizational research, we provide organizational scholars with a step-by-step instructional guide to develop immersive vignette studies.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"78 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2024-04-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140608136","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-13DOI: 10.1177/10944281241244760
Kai Liu, Yi Zheng, Daxun Wang, Yan Cai, Yuanyuan Shi, Chongqin Xi, Dongbo Tu
In recent decades, multidimensional forced-choice (MFC) tests have gained widespread popularity in organizational settings due to their effectiveness in reducing response biases. Detecting differential item functioning (DIF) is crucial in developing MFC tests, as it relates to test fairness and validity. However, existing methods appear insufficient for detecting DIF induced by the interaction between multiple covariates. Furthermore, for multi-category, ordered or continuous covariates, existing approaches often dichotomize them using a-priori cutoffs, commonly using the median of the covariates. This may lead to information loss and reduced power in detecting MFC DIF. To address these limitations, we propose a method to identify both main effect DIF and interactive DIF. This method can automatically search for the optimal cutoffs for ordered or continuous covariates without pre-defined cutoffs. We introduce the rationale behind the proposed method and evaluate its performance through three Monte Carlo simulation studies. Results demonstrate that the proposed method effectively identifies various DIF forms in MFC tests, thereby increasing detection power. Finally, we provide an empirical application to illustrate the practical applicability of the proposed method.
{"title":"A Framework for Detecting Both Main Effect and Interactive DIF in Multidimensional Forced-Choice Assessments","authors":"Kai Liu, Yi Zheng, Daxun Wang, Yan Cai, Yuanyuan Shi, Chongqin Xi, Dongbo Tu","doi":"10.1177/10944281241244760","DOIUrl":"https://doi.org/10.1177/10944281241244760","url":null,"abstract":"In recent decades, multidimensional forced-choice (MFC) tests have gained widespread popularity in organizational settings due to their effectiveness in reducing response biases. Detecting differential item functioning (DIF) is crucial in developing MFC tests, as it relates to test fairness and validity. However, existing methods appear insufficient for detecting DIF induced by the interaction between multiple covariates. Furthermore, for multi-category, ordered or continuous covariates, existing approaches often dichotomize them using a-priori cutoffs, commonly using the median of the covariates. This may lead to information loss and reduced power in detecting MFC DIF. To address these limitations, we propose a method to identify both main effect DIF and interactive DIF. This method can automatically search for the optimal cutoffs for ordered or continuous covariates without pre-defined cutoffs. We introduce the rationale behind the proposed method and evaluate its performance through three Monte Carlo simulation studies. Results demonstrate that the proposed method effectively identifies various DIF forms in MFC tests, thereby increasing detection power. Finally, we provide an empirical application to illustrate the practical applicability of the proposed method.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"9 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2024-04-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140551921","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-01-31DOI: 10.1177/10944281241228186
Jaewoo Jung, Wenjun Zhou, Anne D. Smith
Text analysis, particularly custom dictionaries and topic modeling, has helped advance management and organization theory. Custom dictionaries involve creating word lists to quantify patterns and infer constructs, while topic modeling extracts themes from textual documents to help understand a theoretical domain. Building on these two approaches, we propose another text analysis approach called word-text-topic extraction (WTT), which enhances the efficiency and relevance of text analysis for the sake of theoretical advancement. Specifically, we first identify relevant words for a researcher's theoretical area of interest using word-embedding algorithms. That step is followed by extracting text segments from the textual corpus using a collocation process. Finally, topic modeling is applied to capture themes relevant to the specific theoretical area of interest. To illustrate the WTT approach, we explored one research area needing further theory development—innovation. Using 841 CEOs’ letters to shareholders, we found that our WTT approach provides nuanced features of innovation that differ across industry contexts. We guide researchers on decisions and considerations related to the WTT approach in order to facilitate its use in future studies.
{"title":"From Textual Data to Theoretical Insights: Introducing and Applying the Word-Text-Topic Extraction Approach","authors":"Jaewoo Jung, Wenjun Zhou, Anne D. Smith","doi":"10.1177/10944281241228186","DOIUrl":"https://doi.org/10.1177/10944281241228186","url":null,"abstract":"Text analysis, particularly custom dictionaries and topic modeling, has helped advance management and organization theory. Custom dictionaries involve creating word lists to quantify patterns and infer constructs, while topic modeling extracts themes from textual documents to help understand a theoretical domain. Building on these two approaches, we propose another text analysis approach called word-text-topic extraction (WTT), which enhances the efficiency and relevance of text analysis for the sake of theoretical advancement. Specifically, we first identify relevant words for a researcher's theoretical area of interest using word-embedding algorithms. That step is followed by extracting text segments from the textual corpus using a collocation process. Finally, topic modeling is applied to capture themes relevant to the specific theoretical area of interest. To illustrate the WTT approach, we explored one research area needing further theory development—innovation. Using 841 CEOs’ letters to shareholders, we found that our WTT approach provides nuanced features of innovation that differ across industry contexts. We guide researchers on decisions and considerations related to the WTT approach in order to facilitate its use in future studies.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"99 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2024-01-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139938974","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-01-08DOI: 10.1177/10944281231223412
Balázs Kovács
Organizational research increasingly relies on online review data to gauge perceived valuation and reputation of organizations and products. Online review platforms typically collect ordinal ratings (e.g., 1 to 5 stars); however, researchers often treat them as a cardinal data, calculating aggregate statistics such as the average, the median, or the variance of ratings. In calculating these statistics, ratings are implicitly assumed to be equidistant. We test whether star ratings are equidistant using reviews from two large-scale online review platforms: Amazon.com and Yelp.com. We develop a deep learning framework to analyze the text of the reviews in order to assess their overall valuation. We find that 4 and 5-star ratings, as well as 1 and 2-star ratings, are closer to each other than 3-star ratings are to 2 and 4-star ratings. An additional online experiment corroborates this pattern. Using simulations, we show that the distortion by non-equidistant ratings is especially harmful in cases when organizations receive only a few reviews and when researchers are interested in estimating variance effects. We discuss potential solutions to solve the issue with rating non-equidistance.
{"title":"Five Is the Brightest Star. But by how Much? Testing the Equidistance of Star Ratings in Online Reviews","authors":"Balázs Kovács","doi":"10.1177/10944281231223412","DOIUrl":"https://doi.org/10.1177/10944281231223412","url":null,"abstract":"Organizational research increasingly relies on online review data to gauge perceived valuation and reputation of organizations and products. Online review platforms typically collect ordinal ratings (e.g., 1 to 5 stars); however, researchers often treat them as a cardinal data, calculating aggregate statistics such as the average, the median, or the variance of ratings. In calculating these statistics, ratings are implicitly assumed to be equidistant. We test whether star ratings are equidistant using reviews from two large-scale online review platforms: Amazon.com and Yelp.com. We develop a deep learning framework to analyze the text of the reviews in order to assess their overall valuation. We find that 4 and 5-star ratings, as well as 1 and 2-star ratings, are closer to each other than 3-star ratings are to 2 and 4-star ratings. An additional online experiment corroborates this pattern. Using simulations, we show that the distortion by non-equidistant ratings is especially harmful in cases when organizations receive only a few reviews and when researchers are interested in estimating variance effects. We discuss potential solutions to solve the issue with rating non-equidistance.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"22 12","pages":""},"PeriodicalIF":9.5,"publicationDate":"2024-01-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139446588","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：