Pub Date : 2018-10-01DOI: 10.1027/1614-2241/a000157
Sandra Liliana Camargo, A. Herrera, A. Traynor
The purpose of this work is to identify issues regarding the concept of validity in educational and psychological testing on which there is, and is not, consensus among experts, using an online Delphi study. Although many theorists have expressed their views about the proper characterization of validity, it is important to systematically collect ideas about each aspect of validity. Study participants were recognized academic experts who have led the discussion on the concept of validity in publications and academic meetings in Europe and the United States during recent decades. The Delphi study’s results identify some aspects of the concept of validity, the Standards (2014), and validation about which experts are at an impasse, and others about which consensus can be reached. Based on our findings, some recommendations to advance the conceptualization of validity are offered.
{"title":"Looking for a Consensus in the Discussion About the Concept of Validity: A Delphi Study","authors":"Sandra Liliana Camargo, A. Herrera, A. Traynor","doi":"10.1027/1614-2241/a000157","DOIUrl":"https://doi.org/10.1027/1614-2241/a000157","url":null,"abstract":"The purpose of this work is to identify issues regarding the concept of validity in educational and psychological testing on which there is, and is not, consensus among experts, using an online Delphi study. Although many theorists have expressed their views about the proper characterization of validity, it is important to systematically collect ideas about each aspect of validity. Study participants were recognized academic experts who have led the discussion on the concept of validity in publications and academic meetings in Europe and the United States during recent decades. The Delphi study’s results identify some aspects of the concept of validity, the Standards (2014), and validation about which experts are at an impasse, and others about which consensus can be reached. Based on our findings, some recommendations to advance the conceptualization of validity are offered.","PeriodicalId":18476,"journal":{"name":"Methodology: European Journal of Research Methods for The Behavioral and Social Sciences","volume":"14 1","pages":"146–155"},"PeriodicalIF":3.1,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44361970","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-07-01DOI: 10.1027/1614-2241/a000151
Jan Karem Höhne, M. Revilla, Timo Lenzner
The use of agree/disagree (A/D) questions is a common technique to measure attitudes. For instance, this question format is employed frequently in the Eurobarometer and International Social Survey Programme (ISSP). Theoretical considerations, however, suggest that A/D questions require a complex processing. Therefore, many survey researchers have recommended the use of item-specific (IS) questions, since they seem to be less burdensome. Parallel to this methodological discussion is the discussion around the use of mobile devices for responding to surveys. However, until now, evidence has been lacking as to whether the use of mobile devices for survey response affects the performance of established question formats. In this study, implemented in the Netquest panel in Spain (N = 1,476), we investigated the cognitive effort and response quality associated with A/D and IS questions across PCs and smartphones. For this purpose, we applied a split-ballot design defined by device type and question format. Our analyses revealed longer response times for IS questions than A/D questions, irrespective of the device type and scale length. Also, the IS questions produced better response quality than their A/D counterparts. All in all, the findings indicate a more conscientious response to IS questions compared to A/D questions.
{"title":"Comparing the Performance of Agree/Disagree and Item-Specific Questions Across PCs and Smartphones","authors":"Jan Karem Höhne, M. Revilla, Timo Lenzner","doi":"10.1027/1614-2241/a000151","DOIUrl":"https://doi.org/10.1027/1614-2241/a000151","url":null,"abstract":"The use of agree/disagree (A/D) questions is a common technique to measure attitudes. For instance, this question format is employed frequently in the Eurobarometer and International Social Survey Programme (ISSP). Theoretical considerations, however, suggest that A/D questions require a complex processing. Therefore, many survey researchers have recommended the use of item-specific (IS) questions, since they seem to be less burdensome. Parallel to this methodological discussion is the discussion around the use of mobile devices for responding to surveys. However, until now, evidence has been lacking as to whether the use of mobile devices for survey response affects the performance of established question formats. In this study, implemented in the Netquest panel in Spain (N = 1,476), we investigated the cognitive effort and response quality associated with A/D and IS questions across PCs and smartphones. For this purpose, we applied a split-ballot design defined by device type and question format. Our analyses revealed longer response times for IS questions than A/D questions, irrespective of the device type and scale length. Also, the IS questions produced better response quality than their A/D counterparts. All in all, the findings indicate a more conscientious response to IS questions compared to A/D questions.","PeriodicalId":18476,"journal":{"name":"Methodology: European Journal of Research Methods for The Behavioral and Social Sciences","volume":"14 1","pages":"109–118"},"PeriodicalIF":3.1,"publicationDate":"2018-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48970377","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-07-01DOI: 10.1027/1614-2241/a000150
S. Nestler, K. Geukes, M. Back
The mixed-effects location scale model is an extension of a multilevel model for longitudinal data. It allows covariates to affect both the within-subject variance and the between-subject variance (i.e., the intercept variance) beyond their influence on the means. Typically, the model is applied to two-level data (e.g., the repeated measurements of persons), although researchers are often faced with three-level data (e.g., the repeated measurements of persons within specific situations). Here, we describe an extension of the two-level mixed-effects location scale model to such three-level data. Furthermore, we show how the suggested model can be estimated with Bayesian software, and we present the results of a small simulation study that was conducted to investigate the statistical properties of the suggested approach. Finally, we illustrate the approach by presenting an example from a psychological study that employed ecological momentary assessment.
{"title":"Modeling Intraindividual Variability in Three-Level Multilevel Models","authors":"S. Nestler, K. Geukes, M. Back","doi":"10.1027/1614-2241/a000150","DOIUrl":"https://doi.org/10.1027/1614-2241/a000150","url":null,"abstract":"The mixed-effects location scale model is an extension of a multilevel model for longitudinal data. It allows covariates to affect both the within-subject variance and the between-subject variance (i.e., the intercept variance) beyond their influence on the means. Typically, the model is applied to two-level data (e.g., the repeated measurements of persons), although researchers are often faced with three-level data (e.g., the repeated measurements of persons within specific situations). Here, we describe an extension of the two-level mixed-effects location scale model to such three-level data. Furthermore, we show how the suggested model can be estimated with Bayesian software, and we present the results of a small simulation study that was conducted to investigate the statistical properties of the suggested approach. Finally, we illustrate the approach by presenting an example from a psychological study that employed ecological momentary assessment.","PeriodicalId":18476,"journal":{"name":"Methodology: European Journal of Research Methods for The Behavioral and Social Sciences","volume":"34 8","pages":"95–108"},"PeriodicalIF":3.1,"publicationDate":"2018-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41266862","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-06-21DOI: 10.1027/1614-2241/a000145
M. Moerbeek, L. Hesen
In a discrete-time survival model the occurrence of some event is measured by the end of each time interval. In practice it is not always possible to measure all subjects at the same point in time. In this study the consequences of varying measurement occasions are investigated by means of a simulation study and the analysis of data from an empirical study. The results of the simulation study suggest that the effects of varying measurement occasions are negligible, at least for the scenarios that were covered in the simulation. The empirical example shows varying measurement occasions have minor effects on parameter estimates, standard errors, and significance levels.
{"title":"The Consequences of Varying Measurement Occasions in Discrete-Time Survival Analysis","authors":"M. Moerbeek, L. Hesen","doi":"10.1027/1614-2241/a000145","DOIUrl":"https://doi.org/10.1027/1614-2241/a000145","url":null,"abstract":"In a discrete-time survival model the occurrence of some event is measured by the end of each time interval. In practice it is not always possible to measure all subjects at the same point in time. In this study the consequences of varying measurement occasions are investigated by means of a simulation study and the analysis of data from an empirical study. The results of the simulation study suggest that the effects of varying measurement occasions are negligible, at least for the scenarios that were covered in the simulation. The empirical example shows varying measurement occasions have minor effects on parameter estimates, standard errors, and significance levels.","PeriodicalId":18476,"journal":{"name":"Methodology: European Journal of Research Methods for The Behavioral and Social Sciences","volume":"14 1","pages":"45–55"},"PeriodicalIF":3.1,"publicationDate":"2018-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47659661","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-06-21DOI: 10.1027/1614-2241/a000148
Yi-Jhen Wu, Insu Paek
When using the mixture Rasch model, the model identification constraints are either to set the equal means for all classes in the assumed normal ability distributions (equal ability mean constraint in short), or to set the sum of item difficulties to be zero for each class. In real data analysis, however, both constraints are not always sufficient to establish a common scale across latent classes unless some items are specified as anchor items in the estimation. If these two conventional constraint approaches recover the class membership as good as the anchor item constraint approach, the conventional constraint approaches may be considered useful for the purpose of class membership classification. This study investigated agreement on class membership between one conventional constraint (the equal ability mean) and the anchor item constraint approaches. Results showed high agreement between these two constraint approaches, indicating that the conventional constraint of the equal mean ability approach may be used to recover the latent class membership although item profiles are not correctly estimated across latent classes.
{"title":"Agreement on the Classification of Latent Class Membership Between Different Identification Constraint Approaches in the Mixture Rasch Model","authors":"Yi-Jhen Wu, Insu Paek","doi":"10.1027/1614-2241/a000148","DOIUrl":"https://doi.org/10.1027/1614-2241/a000148","url":null,"abstract":"When using the mixture Rasch model, the model identification constraints are either to set the equal means for all classes in the assumed normal ability distributions (equal ability mean constraint in short), or to set the sum of item difficulties to be zero for each class. In real data analysis, however, both constraints are not always sufficient to establish a common scale across latent classes unless some items are specified as anchor items in the estimation. If these two conventional constraint approaches recover the class membership as good as the anchor item constraint approach, the conventional constraint approaches may be considered useful for the purpose of class membership classification. This study investigated agreement on class membership between one conventional constraint (the equal ability mean) and the anchor item constraint approaches. Results showed high agreement between these two constraint approaches, indicating that the conventional constraint of the equal mean ability approach may be used to recover the latent class membership although item profiles are not correctly estimated across latent classes.","PeriodicalId":18476,"journal":{"name":"Methodology: European Journal of Research Methods for The Behavioral and Social Sciences","volume":"14 1","pages":"82–93"},"PeriodicalIF":3.1,"publicationDate":"2018-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43688635","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-06-21DOI: 10.1027/1614-2241/a000147
J. Cremers, T. Mainhard, I. Klugkist
Circular data is different from linear data and its analysis also requires methods different from conventional methods. In this study a Bayesian embedding approach to estimating circular regression models is investigated, by means of simulation studies, in terms of performance, efficiency, and flexibility. A new Markov chain Monte Carlo (MCMC) sampling method is proposed and contrasted to an existing method. An empirical example of a regression model predicting teachers’ scores on the interpersonal circumplex will be used throughout. Performance and efficiency are better for the newly proposed sampler and reasonable to good in most situations. Furthermore, the method in general is deemed very flexible. Additional research should be done that provides an overview of what circular data looks like in practice, investigates the interpretation of the circular effects and examines how we might conduct a way of hypothesis testing or model checking for the embedding approach.
{"title":"Assessing a Bayesian Embedding Approach to Circular Regression Models","authors":"J. Cremers, T. Mainhard, I. Klugkist","doi":"10.1027/1614-2241/a000147","DOIUrl":"https://doi.org/10.1027/1614-2241/a000147","url":null,"abstract":"Circular data is different from linear data and its analysis also requires methods different from conventional methods. In this study a Bayesian embedding approach to estimating circular regression models is investigated, by means of simulation studies, in terms of performance, efficiency, and flexibility. A new Markov chain Monte Carlo (MCMC) sampling method is proposed and contrasted to an existing method. An empirical example of a regression model predicting teachers’ scores on the interpersonal circumplex will be used throughout. Performance and efficiency are better for the newly proposed sampler and reasonable to good in most situations. Furthermore, the method in general is deemed very flexible. Additional research should be done that provides an overview of what circular data looks like in practice, investigates the interpretation of the circular effects and examines how we might conduct a way of hypothesis testing or model checking for the embedding approach.","PeriodicalId":18476,"journal":{"name":"Methodology: European Journal of Research Methods for The Behavioral and Social Sciences","volume":"14 1","pages":"69–81"},"PeriodicalIF":3.1,"publicationDate":"2018-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44890967","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-06-21DOI: 10.1027/1614-2241/a000146
D. Vidotto, J. Vermunt, K. Van Deun
Latent class analysis has been recently proposed for the multiple imputation (MI) of missing categorical data, using either a standard frequentist approach or a nonparametric Bayesian model called Dirichlet process mixture of multinomial distributions (DPMM). The main advantage of using a latent class model for multiple imputation is that it is very flexible in the sense that it can capture complex relationships in the data given that the number of latent classes is large enough. However, the two existing approaches also have certain disadvantages. The frequentist approach is computationally demanding because it requires estimating many LC models: first models with different number of classes should be estimated to determine the required number of classes and subsequently the selected model is reestimated for multiple bootstrap samples to take into account parameter uncertainty during the imputation stage. Whereas the Bayesian Dirichlet process models perform the model selection and the handling of the parameter uncertainty automatically, the disadvantage of this method is that it tends to use a too small number of clusters during the Gibbs sampling, leading to an underfitting model yielding invalid imputations. In this paper, we propose an alternative approach which combined the strengths of the two existing approaches; that is, we use the Bayesian standard latent class model as an imputation model. We show how model selection can be performed prior to the imputation step using a single run of the Gibbs sampler and, moreover, show how underfitting is prevented by using large values for the hyperparameters of the mixture weights. The results of two simulation studies and one real-data study indicate that with a proper setting of the prior distributions, the Bayesian latent class model yields valid imputations and outperforms competing methods.
{"title":"Bayesian Latent Class Models for the Multiple Imputation of Categorical Data","authors":"D. Vidotto, J. Vermunt, K. Van Deun","doi":"10.1027/1614-2241/a000146","DOIUrl":"https://doi.org/10.1027/1614-2241/a000146","url":null,"abstract":"Latent class analysis has been recently proposed for the multiple imputation (MI) of missing categorical data, using either a standard frequentist approach or a nonparametric Bayesian model called Dirichlet process mixture of multinomial distributions (DPMM). The main advantage of using a latent class model for multiple imputation is that it is very flexible in the sense that it can capture complex relationships in the data given that the number of latent classes is large enough. However, the two existing approaches also have certain disadvantages. The frequentist approach is computationally demanding because it requires estimating many LC models: first models with different number of classes should be estimated to determine the required number of classes and subsequently the selected model is reestimated for multiple bootstrap samples to take into account parameter uncertainty during the imputation stage. Whereas the Bayesian Dirichlet process models perform the model selection and the handling of the parameter uncertainty automatically, the disadvantage of this method is that it tends to use a too small number of clusters during the Gibbs sampling, leading to an underfitting model yielding invalid imputations. In this paper, we propose an alternative approach which combined the strengths of the two existing approaches; that is, we use the Bayesian standard latent class model as an imputation model. We show how model selection can be performed prior to the imputation step using a single run of the Gibbs sampler and, moreover, show how underfitting is prevented by using large values for the hyperparameters of the mixture weights. The results of two simulation studies and one real-data study indicate that with a proper setting of the prior distributions, the Bayesian latent class model yields valid imputations and outperforms competing methods.","PeriodicalId":18476,"journal":{"name":"Methodology: European Journal of Research Methods for The Behavioral and Social Sciences","volume":"14 1","pages":"56–68"},"PeriodicalIF":3.1,"publicationDate":"2018-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41370291","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-04-23DOI: 10.1027/1614-2241/a000141
Kristian Kleinke
Predictive mean matching (PMM) is a state-of-the-art hot deck multiple imputation (MI) procedure. The quality of its results depends, inter alia, on the availability of suitable donor cases. Applying PMM in small sample scenarios often found in psychological or medical research could be problematic, as there might not be many (or any) suitable donor cases in the data set. So far, there has not been any systematic research that examined the performance of PMM, when sample size is small. The present study evaluated PMM in various multiple regression scenarios, where sample size, missing data percentages, the size of the regression coefficients, and PMM’s donor selection strategy were systematically varied. Results show that PMM could be used in most scenarios, however results depended on the donor selection strategy: overall, PMM using either automatic distance-aided selection of donors (Gaffert, Meinfelder, & Bosch, 2016) or using the nearest neighbor produced the best results.
{"title":"Multiple Imputation by Predictive Mean Matching When Sample Size Is Small","authors":"Kristian Kleinke","doi":"10.1027/1614-2241/a000141","DOIUrl":"https://doi.org/10.1027/1614-2241/a000141","url":null,"abstract":"Predictive mean matching (PMM) is a state-of-the-art hot deck multiple imputation (MI) procedure. The quality of its results depends, inter alia, on the availability of suitable donor cases. Applying PMM in small sample scenarios often found in psychological or medical research could be problematic, as there might not be many (or any) suitable donor cases in the data set. So far, there has not been any systematic research that examined the performance of PMM, when sample size is small. The present study evaluated PMM in various multiple regression scenarios, where sample size, missing data percentages, the size of the regression coefficients, and PMM’s donor selection strategy were systematically varied. Results show that PMM could be used in most scenarios, however results depended on the donor selection strategy: overall, PMM using either automatic distance-aided selection of donors (Gaffert, Meinfelder, & Bosch, 2016) or using the nearest neighbor produced the best results.","PeriodicalId":18476,"journal":{"name":"Methodology: European Journal of Research Methods for The Behavioral and Social Sciences","volume":"14 1","pages":"3–15"},"PeriodicalIF":3.1,"publicationDate":"2018-04-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48312465","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-04-23DOI: 10.1027/1614-2241/a000143
Audrey J. Leroux, S. Beretvas
The current study proposed a new model, termed the cross-classified multiple membership latent variable regression (CCMM-LVR) model that provides an extension to the three-level latent variable regression (HM3-LVR) model that can be used with cross-classified multiple membership data, for example, in the presence of student mobility across schools. The HM3-LVR model is beneficial for testing more flexible hypotheses about growth trajectory parameters and handles pure clustering of participants within higher-level (level-3) units. However, the HM3-LVR model involves the assumption that students remain in the same cluster (school) throughout the duration of the time period of interest. The CCMM-LVR model appropriately models the participants’ changing clusters over time. The impact of ignoring mobility in the real data was investigated by comparing parameter estimates, standard error estimates, and model fit indices for the model (CCMM-LVR) that appropriately modeled the cross-classified multiple membership structure with results when this structure was ignored (HM3-LVR).
{"title":"Estimating a Three-Level Latent Variable Regression Model With Cross-Classified Multiple Membership Data","authors":"Audrey J. Leroux, S. Beretvas","doi":"10.1027/1614-2241/a000143","DOIUrl":"https://doi.org/10.1027/1614-2241/a000143","url":null,"abstract":"The current study proposed a new model, termed the cross-classified multiple membership latent variable regression (CCMM-LVR) model that provides an extension to the three-level latent variable regression (HM3-LVR) model that can be used with cross-classified multiple membership data, for example, in the presence of student mobility across schools. The HM3-LVR model is beneficial for testing more flexible hypotheses about growth trajectory parameters and handles pure clustering of participants within higher-level (level-3) units. However, the HM3-LVR model involves the assumption that students remain in the same cluster (school) throughout the duration of the time period of interest. The CCMM-LVR model appropriately models the participants’ changing clusters over time. The impact of ignoring mobility in the real data was investigated by comparing parameter estimates, standard error estimates, and model fit indices for the model (CCMM-LVR) that appropriately modeled the cross-classified multiple membership structure with results when this structure was ignored (HM3-LVR).","PeriodicalId":18476,"journal":{"name":"Methodology: European Journal of Research Methods for The Behavioral and Social Sciences","volume":"14 1","pages":"30–44"},"PeriodicalIF":3.1,"publicationDate":"2018-04-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43905996","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2017-11-24DOI: 10.1027/1614-2241/a000137
Alejandro Hernández‐Camacho, J. Olea, F. J. Abad
The bifactor model (BM) and the testlet response model (TRM) are the most common multidimensional models applied to testlet-based tests. The common procedure is to estimate these models using different estimation methods (see, e.g., DeMars, 2006). A possible consequence of this is that previous findings about the implications of fitting a wrong model to the data may be confounded with the estimation procedures they employed. With this in mind, the present study uses the same method (maximum marginal likelihood [MML] using dimensional reduction) to compare uni- and multidimensional strategies to testlet-based tests, and assess the performance of various relative fit indices. Data were simulated under three different models, namely BM, TRM, and the unidimensional model. Recovery of item parameters, reliability estimates, and selection rates of the relative fit indices were documented. The results were essentially consistent with those obtained through different methods (DeMars, 2006), indicating that the effect of the estimation method is negligible. Regarding the fit indices, Akaike Information Criterion (AIC) showed the best selection rates, whereas Bayes Information Criterion (BIC) tended to select a model which is simpler than the true one. The work concludes with recommendations for practitioners and proposals for future research.
{"title":"Comparison of Uni- and Multidimensional Models Applied in Testlet-Based Tests","authors":"Alejandro Hernández‐Camacho, J. Olea, F. J. Abad","doi":"10.1027/1614-2241/a000137","DOIUrl":"https://doi.org/10.1027/1614-2241/a000137","url":null,"abstract":"The bifactor model (BM) and the testlet response model (TRM) are the most common multidimensional models applied to testlet-based tests. The common procedure is to estimate these models using different estimation methods (see, e.g., DeMars, 2006). A possible consequence of this is that previous findings about the implications of fitting a wrong model to the data may be confounded with the estimation procedures they employed. With this in mind, the present study uses the same method (maximum marginal likelihood [MML] using dimensional reduction) to compare uni- and multidimensional strategies to testlet-based tests, and assess the performance of various relative fit indices. Data were simulated under three different models, namely BM, TRM, and the unidimensional model. Recovery of item parameters, reliability estimates, and selection rates of the relative fit indices were documented. The results were essentially consistent with those obtained through different methods (DeMars, 2006), indicating that the effect of the estimation method is negligible. Regarding the fit indices, Akaike Information Criterion (AIC) showed the best selection rates, whereas Bayes Information Criterion (BIC) tended to select a model which is simpler than the true one. The work concludes with recommendations for practitioners and proposals for future research.","PeriodicalId":18476,"journal":{"name":"Methodology: European Journal of Research Methods for The Behavioral and Social Sciences","volume":"13 1","pages":"135–143"},"PeriodicalIF":3.1,"publicationDate":"2017-11-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42189419","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：