Pub Date : 2023-05-30DOI: 10.1080/15305058.2023.2214648
Fatima Munevver Saatcioglu, Sedat Sen
AbstractIn this study, we illustrated an application of the confirmatory mixture IRT model for multidimensional tests. We aimed to examine the differences in student performance by domains with a confirmatory mixture IRT modeling approach. A three-dimensional and three-class model was analyzed by assuming content domains as dimensions and cognitive domains as item groups. We estimated the item performance differences among the students through structural parameters. There were 463 students from Turkey and 880 students from Canada who participated in the TIMSS 2015 4th-grade mathematics assessment. Results for Turkey indicated, students in Class 2 had better performance in knowing and reasoning compared to those in Classes 1 and 3. Students in Class 2 and Class 3 were similar in applying math concepts compared to students in Class 1. For the Canadian sample, students in Class 2 had better performance in knowing, applying, and reasoning compared to those in Class 1 and 3. Also, Class 3 students were better at applying domain than Class 1. Also, mean values were obtained for all content domains in the two countries. Confirmatory mixture IRT modeling approaches appear to differentiate students’ mathematics competencies.Keywords: Confirmatory mixture IRT modelinglatent classmultidimensional modelTIMSS 2015
{"title":"The analysis of TIMSS 2015 data with confirmatory mixture item response theory: A multidimensional approach","authors":"Fatima Munevver Saatcioglu, Sedat Sen","doi":"10.1080/15305058.2023.2214648","DOIUrl":"https://doi.org/10.1080/15305058.2023.2214648","url":null,"abstract":"AbstractIn this study, we illustrated an application of the confirmatory mixture IRT model for multidimensional tests. We aimed to examine the differences in student performance by domains with a confirmatory mixture IRT modeling approach. A three-dimensional and three-class model was analyzed by assuming content domains as dimensions and cognitive domains as item groups. We estimated the item performance differences among the students through structural parameters. There were 463 students from Turkey and 880 students from Canada who participated in the TIMSS 2015 4th-grade mathematics assessment. Results for Turkey indicated, students in Class 2 had better performance in knowing and reasoning compared to those in Classes 1 and 3. Students in Class 2 and Class 3 were similar in applying math concepts compared to students in Class 1. For the Canadian sample, students in Class 2 had better performance in knowing, applying, and reasoning compared to those in Class 1 and 3. Also, Class 3 students were better at applying domain than Class 1. Also, mean values were obtained for all content domains in the two countries. Confirmatory mixture IRT modeling approaches appear to differentiate students’ mathematics competencies.Keywords: Confirmatory mixture IRT modelinglatent classmultidimensional modelTIMSS 2015","PeriodicalId":46615,"journal":{"name":"International Journal of Testing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-05-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135643183","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-04-18DOI: 10.1080/15305058.2023.2195662
Masoomeh Estaji, Zahra Banitalebi
Abstract This study used Latent Growth Curve Modeling (LGCM) to examine the overtime patterns of the score and test-taking strategy changes in an international high-stakes standardized proficiency test. To this end, the test records of 178 Iranian IELTS repeaters were analyzed, using close- and open-ended questionnaires to measure test scores as a function of construct-relevant and construct-irrelevant test-taking strategy changes. Additionally, this study explored the accountable factors for the changes in the repeaters’ strategies. Results indicated a small and gradual increase in the test scores following an overall augmented use of test-management (TM) and a decreased employment of test-wiseness (TW) strategies. Along with contributing to IELTS validity evidence based on the repeaters’ scores, this study found multiple sources to account for the changes in repeaters’ test-taking strategies. Consideration of changes in repeaters’ test-taking strategies by IELTS instructors and test users may add to the validity of interpretation of test scores to the intended purposes of the tests.
{"title":"A study of Test-Taking strategies of Iranian IELTS repeaters: Any change in the strategy use?","authors":"Masoomeh Estaji, Zahra Banitalebi","doi":"10.1080/15305058.2023.2195662","DOIUrl":"https://doi.org/10.1080/15305058.2023.2195662","url":null,"abstract":"Abstract This study used Latent Growth Curve Modeling (LGCM) to examine the overtime patterns of the score and test-taking strategy changes in an international high-stakes standardized proficiency test. To this end, the test records of 178 Iranian IELTS repeaters were analyzed, using close- and open-ended questionnaires to measure test scores as a function of construct-relevant and construct-irrelevant test-taking strategy changes. Additionally, this study explored the accountable factors for the changes in the repeaters’ strategies. Results indicated a small and gradual increase in the test scores following an overall augmented use of test-management (TM) and a decreased employment of test-wiseness (TW) strategies. Along with contributing to IELTS validity evidence based on the repeaters’ scores, this study found multiple sources to account for the changes in repeaters’ test-taking strategies. Consideration of changes in repeaters’ test-taking strategies by IELTS instructors and test users may add to the validity of interpretation of test scores to the intended purposes of the tests.","PeriodicalId":46615,"journal":{"name":"International Journal of Testing","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-04-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44558305","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-04-18DOI: 10.1080/15305058.2023.2195659
Nico Streckert, Lara Kurtz, P. Kajonius
Abstract The Dark Factor of Personality (D) measures the latent core of antagonistic traits. The present study evaluated the psychometric properties of the Swedish version of the full (D70) and the brief (D16) versions, concerning structural validity, item information, and convergent validity. An online sample (N = 294) was analyzed using CFA (Maximum Likelihood Estimation), IRT (Graded Response Model) and SEM (latent correlations). Firstly, the original theorized bifactor model for D70 and a single-factor model for D16 showed good fit to the data. Moreover, new reliability-analyses based on FD and H indicated that the D70 favorably can be collapsed into a unidimensional measure, which is further discussed. Secondly, the IRT-analyses present valid item quality and functioning and showed that items provide the most information on trait levels above mean levels. Lastly, convergent SEM-analyses showed that D had high latent trait correlations to psychopathy and Machiavellianism, but not to narcissism. The correlations with the Big Six personality factors (mini-IPIP6) yielded expected high correlations with Agreeableness and Honesty-Humility. The Swedish translation of the full D70 and brief D16 is recommended for use in future research.
{"title":"Can your darkness be measured? Analyzing the full and brief version of the Dark Factor of Personality in Swedish","authors":"Nico Streckert, Lara Kurtz, P. Kajonius","doi":"10.1080/15305058.2023.2195659","DOIUrl":"https://doi.org/10.1080/15305058.2023.2195659","url":null,"abstract":"Abstract The Dark Factor of Personality (D) measures the latent core of antagonistic traits. The present study evaluated the psychometric properties of the Swedish version of the full (D70) and the brief (D16) versions, concerning structural validity, item information, and convergent validity. An online sample (N = 294) was analyzed using CFA (Maximum Likelihood Estimation), IRT (Graded Response Model) and SEM (latent correlations). Firstly, the original theorized bifactor model for D70 and a single-factor model for D16 showed good fit to the data. Moreover, new reliability-analyses based on FD and H indicated that the D70 favorably can be collapsed into a unidimensional measure, which is further discussed. Secondly, the IRT-analyses present valid item quality and functioning and showed that items provide the most information on trait levels above mean levels. Lastly, convergent SEM-analyses showed that D had high latent trait correlations to psychopathy and Machiavellianism, but not to narcissism. The correlations with the Big Six personality factors (mini-IPIP6) yielded expected high correlations with Agreeableness and Honesty-Humility. The Swedish translation of the full D70 and brief D16 is recommended for use in future research.","PeriodicalId":46615,"journal":{"name":"International Journal of Testing","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-04-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45280757","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-04-03DOI: 10.1080/15305058.2023.2195661
Carla Martí Valls, Kitty Balazadeh, P. Kajonius
Abstract The Alternative DSM-5 Model for Personality Disorders (AMPD) consists of level of personality functioning (Criterion A) and maladaptive personality traits (Criterion B). The brief scale versions of these are understudied, while often being used by clinicians and researchers. In this study, we wanted to investigate the overlap and predictive validity of Criterion A and B. Participants (N = 253) were measured on level of personality functioning (LPFS-BF) and maladaptive personality traits (PID-5-BF), as well as internalizing outcomes such existential meaninglessness (EMS) and externalizing outcomes such as substance and behavioral addictions (SSAB). Data analysis was conducted with principal component analysis (PCA) and regression analyses. The results showed over 50% overlap between the brief versions of Criterion A and B, while Criterion B slightly outperformed Criterion A in outcomes of EMS and SSAB. We discuss the potential redundancy and usefulness of personality functioning and maladaptive personality traits.
{"title":"Investigating the overlap and predictive validity between Criterion A and B in the alternative model for personality disorders in DSM-5","authors":"Carla Martí Valls, Kitty Balazadeh, P. Kajonius","doi":"10.1080/15305058.2023.2195661","DOIUrl":"https://doi.org/10.1080/15305058.2023.2195661","url":null,"abstract":"Abstract The Alternative DSM-5 Model for Personality Disorders (AMPD) consists of level of personality functioning (Criterion A) and maladaptive personality traits (Criterion B). The brief scale versions of these are understudied, while often being used by clinicians and researchers. In this study, we wanted to investigate the overlap and predictive validity of Criterion A and B. Participants (N = 253) were measured on level of personality functioning (LPFS-BF) and maladaptive personality traits (PID-5-BF), as well as internalizing outcomes such existential meaninglessness (EMS) and externalizing outcomes such as substance and behavioral addictions (SSAB). Data analysis was conducted with principal component analysis (PCA) and regression analyses. The results showed over 50% overlap between the brief versions of Criterion A and B, while Criterion B slightly outperformed Criterion A in outcomes of EMS and SSAB. We discuss the potential redundancy and usefulness of personality functioning and maladaptive personality traits.","PeriodicalId":46615,"journal":{"name":"International Journal of Testing","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48832344","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-01-18DOI: 10.1080/15305058.2023.2167084
Rongxiu Wu, C. Chiu, David M. Dueber, Mirang Park, D. Lange, Emre Umucu, D. Strauser
Abstract The current study examined the factor structure, measurement invariance, and construct validity of the 14-item Revised Developmental Work Personality Scale (RDWPS) using a sample of 603 college students in a Midwest university of the United States. Exploratory and confirmatory factor analysis results indicated that the 11-item RDWPS resulted in a better fit of the measurement model. Partial measurement invariance was also detected between gender groups. In addition, it was weakly to moderately correlated with the Utrecht Work Engagement Scale-Student (UWES-S), self-reported effort, and GPA among college students. Lastly, it was found that males scored lower than females in all three subscales of the RDWPS in comparison to the latent means of the gender groups.
{"title":"Multidimensionality and measurement invariance of the revised developmental work personality scale","authors":"Rongxiu Wu, C. Chiu, David M. Dueber, Mirang Park, D. Lange, Emre Umucu, D. Strauser","doi":"10.1080/15305058.2023.2167084","DOIUrl":"https://doi.org/10.1080/15305058.2023.2167084","url":null,"abstract":"Abstract The current study examined the factor structure, measurement invariance, and construct validity of the 14-item Revised Developmental Work Personality Scale (RDWPS) using a sample of 603 college students in a Midwest university of the United States. Exploratory and confirmatory factor analysis results indicated that the 11-item RDWPS resulted in a better fit of the measurement model. Partial measurement invariance was also detected between gender groups. In addition, it was weakly to moderately correlated with the Utrecht Work Engagement Scale-Student (UWES-S), self-reported effort, and GPA among college students. Lastly, it was found that males scored lower than females in all three subscales of the RDWPS in comparison to the latent means of the gender groups.","PeriodicalId":46615,"journal":{"name":"International Journal of Testing","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-01-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49347768","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-12-28DOI: 10.1080/15305058.2022.2149536
L.H.L. Badham, Antony Furlong
Abstract Multilingual summative assessments face significant challenges due to tensions that exist between multiple language provision and comparability. Yet, conventional approaches for investigating comparability in multilingual assessments fail to accommodate assessments that comprise extended responses that target complex constructs. This article discusses a study that investigated whether bilingual examiners could apply comparative judgment (CJ) to pairs of Literature essays across different languages (English and Spanish). Preliminary findings suggest that whilst there are some cross-language standardization benefits, bilingual CJ faces validity challenges when different language cohorts approach target constructs differently. Existing definitions of inter-subject and intra-subject comparability are insufficient when multilingual subjects share fundamental constructs but differ in academic approaches. It is therefore proposed that an overarching classification of intra-disciplinary comparability be introduced to frame discussions around multilingual assessments of this nature. Finally, it is recommended that further research into bilingual CJ be carried out to determine how the method can most effectively support investigations into multilingual assessment comparability.
{"title":"Summative assessments in a multilingual context: What comparative judgment reveals about comparability across different languages in Literature","authors":"L.H.L. Badham, Antony Furlong","doi":"10.1080/15305058.2022.2149536","DOIUrl":"https://doi.org/10.1080/15305058.2022.2149536","url":null,"abstract":"Abstract Multilingual summative assessments face significant challenges due to tensions that exist between multiple language provision and comparability. Yet, conventional approaches for investigating comparability in multilingual assessments fail to accommodate assessments that comprise extended responses that target complex constructs. This article discusses a study that investigated whether bilingual examiners could apply comparative judgment (CJ) to pairs of Literature essays across different languages (English and Spanish). Preliminary findings suggest that whilst there are some cross-language standardization benefits, bilingual CJ faces validity challenges when different language cohorts approach target constructs differently. Existing definitions of inter-subject and intra-subject comparability are insufficient when multilingual subjects share fundamental constructs but differ in academic approaches. It is therefore proposed that an overarching classification of intra-disciplinary comparability be introduced to frame discussions around multilingual assessments of this nature. Finally, it is recommended that further research into bilingual CJ be carried out to determine how the method can most effectively support investigations into multilingual assessment comparability.","PeriodicalId":46615,"journal":{"name":"International Journal of Testing","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2022-12-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41885703","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-12-19DOI: 10.1080/15305058.2022.2148185
Lucas de Francisco Carvalho, A. Gonçalves, Amanda Rizzieri Romano, Antônio da Conceição Montes, G. Machado, Giselle Pianowski
Abstract We developed and validated a self-report scale for screening pathological traits of dependent personality disorder (DPD) from the Hierarchical Taxonomy of psychopathology (HiTOP) perspective. The sample was 693 adults who answered the new scale, the Dimensional Clinical Personality Inventory DPD (IDCP-DPD), the PID-5, the FFDI, and the FFBI. The IDCP-DPD was composed of six factors grouped in one general score. The scores showed associations with external measures in the expected direction, and the means comparisons showed large differences. Our findings indicated the IDCP-DPD as a useful clinical measure, and the structure observed confirms the spectrum level of the HiTOP.
{"title":"Measuring pathological traits of the dependent personality disorder based on the HiTOP","authors":"Lucas de Francisco Carvalho, A. Gonçalves, Amanda Rizzieri Romano, Antônio da Conceição Montes, G. Machado, Giselle Pianowski","doi":"10.1080/15305058.2022.2148185","DOIUrl":"https://doi.org/10.1080/15305058.2022.2148185","url":null,"abstract":"Abstract We developed and validated a self-report scale for screening pathological traits of dependent personality disorder (DPD) from the Hierarchical Taxonomy of psychopathology (HiTOP) perspective. The sample was 693 adults who answered the new scale, the Dimensional Clinical Personality Inventory DPD (IDCP-DPD), the PID-5, the FFDI, and the FFBI. The IDCP-DPD was composed of six factors grouped in one general score. The scores showed associations with external measures in the expected direction, and the means comparisons showed large differences. Our findings indicated the IDCP-DPD as a useful clinical measure, and the structure observed confirms the spectrum level of the HiTOP.","PeriodicalId":46615,"journal":{"name":"International Journal of Testing","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2022-12-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45945681","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-12-19DOI: 10.1080/15305058.2022.2147938
Lucas de Francisco Carvalho, Camila Grillo Santos, Nelson Fernandes Junior, Rafael Moreton Alves da Rocha, Talita Meireles Flores, Gisele Magarotto Machado
Abstract We aimed to refine the previously proposed antisocial subscale for the Dimensional Clinical Personality Inventory 2 (IDCP-ASPD). The sample involved 628 Brazilian adults between 18 and 81 years old. We administered the revised ASPD subscale (IDCP-ASPD-R), the Affective and Cognitive Measure of Empathy (ACME), the Crime and Analogous Behavior Scale (CAB), and the Levenson Self-Report Psychopathy (LSRP). We confirmed the 3-factors structure for the IDCP-ASPD-R. The IDCP-ASPD-R and its former version presented a good capacity to distinguish the groups, with the largest effect size for the Affective factor (IDCP-ASPD-R). Although the IDCP-ASPD-R has shown good performance, we have observed only a slight increase over the previous version of the scale. Therefore, we can only expect a small higher contribution of IDCP-ASPD-R in its practical application to group discrimination. However, from a theoretical perspective, the IDCP-ASPD-R overrides its former version.
{"title":"Refining the antisocial subscale of the dimensional clinical personality inventory 2: Failed improvements or did we reach the mountain top","authors":"Lucas de Francisco Carvalho, Camila Grillo Santos, Nelson Fernandes Junior, Rafael Moreton Alves da Rocha, Talita Meireles Flores, Gisele Magarotto Machado","doi":"10.1080/15305058.2022.2147938","DOIUrl":"https://doi.org/10.1080/15305058.2022.2147938","url":null,"abstract":"Abstract We aimed to refine the previously proposed antisocial subscale for the Dimensional Clinical Personality Inventory 2 (IDCP-ASPD). The sample involved 628 Brazilian adults between 18 and 81 years old. We administered the revised ASPD subscale (IDCP-ASPD-R), the Affective and Cognitive Measure of Empathy (ACME), the Crime and Analogous Behavior Scale (CAB), and the Levenson Self-Report Psychopathy (LSRP). We confirmed the 3-factors structure for the IDCP-ASPD-R. The IDCP-ASPD-R and its former version presented a good capacity to distinguish the groups, with the largest effect size for the Affective factor (IDCP-ASPD-R). Although the IDCP-ASPD-R has shown good performance, we have observed only a slight increase over the previous version of the scale. Therefore, we can only expect a small higher contribution of IDCP-ASPD-R in its practical application to group discrimination. However, from a theoretical perspective, the IDCP-ASPD-R overrides its former version.","PeriodicalId":46615,"journal":{"name":"International Journal of Testing","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2022-12-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42968250","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-02DOI: 10.1080/15305058.2022.2134394
R. Landers, Elena M. Auer, Gabriel Mersy, Sebastian Marin, Jason Blaik
Abstract Assessment trace data, such as mouse positions and their timing, offer interesting and provocative reflections of individual differences yet are currently underutilized by testing professionals. In this article, we present a 10-step procedure to maximize the probability that a trace data modeling project will be successful: 1) grounding the project in psychometric theory, 2) building technical infrastructure to collect trace data, 3) designing a useful developmental validation study, 4) using a holdout validation approach with collected data, 5) using exploratory analysis to conduct meaningful feature engineering, 6) identifying useful machine learning algorithms to predict a thoughtfully chosen criterion, 7) engineering a machine learning model with meaningful internal cross-validation and hyperparameter selection, 8) conducting model diagnostics to assess if the resulting model is overfitted, underfitted, or within acceptable tolerance, and 9) testing the success of the final model in meeting conceptual, technical, and psychometric goals. If deemed successful, trace data model predictions could then be engineered into decision-making systems. We present this framework within the broader view of psychometrics, exploring the challenges of developing psychometrically valid models using such complex data with much weaker trait signals than assessment developers have typically attempted to model.
{"title":"You are what you click: using machine learning to model trace data for psychometric measurement","authors":"R. Landers, Elena M. Auer, Gabriel Mersy, Sebastian Marin, Jason Blaik","doi":"10.1080/15305058.2022.2134394","DOIUrl":"https://doi.org/10.1080/15305058.2022.2134394","url":null,"abstract":"Abstract Assessment trace data, such as mouse positions and their timing, offer interesting and provocative reflections of individual differences yet are currently underutilized by testing professionals. In this article, we present a 10-step procedure to maximize the probability that a trace data modeling project will be successful: 1) grounding the project in psychometric theory, 2) building technical infrastructure to collect trace data, 3) designing a useful developmental validation study, 4) using a holdout validation approach with collected data, 5) using exploratory analysis to conduct meaningful feature engineering, 6) identifying useful machine learning algorithms to predict a thoughtfully chosen criterion, 7) engineering a machine learning model with meaningful internal cross-validation and hyperparameter selection, 8) conducting model diagnostics to assess if the resulting model is overfitted, underfitted, or within acceptable tolerance, and 9) testing the success of the final model in meeting conceptual, technical, and psychometric goals. If deemed successful, trace data model predictions could then be engineered into decision-making systems. We present this framework within the broader view of psychometrics, exploring the challenges of developing psychometrically valid models using such complex data with much weaker trait signals than assessment developers have typically attempted to model.","PeriodicalId":46615,"journal":{"name":"International Journal of Testing","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2022-10-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49369149","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-10-02DOI: 10.1080/15305058.2022.2036160
Efsun Birtwistle, Ramona Schoedel, Florian Bemmann, Astrid Wirth, Christoph Sürig, Clemens Stachl, M. Bühner, Frank Niklas
Abstract Digital technologies play an important role in our daily lives. Smartphones and tablet computers are very common worldwide and are available for everybody from a very early age. This trend offers the opportunity to track digital usage data for psychological and educational research purposes. The current paper introduces two research projects, the PhoneStudy and Learning4Kids that both use mobile sensing software to collect ecologically valid data on the usage of applications installed on smartphones and tablets. This usage data is used for statistical analyses, for a reward system, and to provide feedback to the study participants. The advantages and challenges of using mobile sensing compared to conventional forms of assessments, and the potential applications of mobile sensing in psychological and educational research are discussed.
{"title":"Mobile sensing in psychological and educational research: Examples from two application fields","authors":"Efsun Birtwistle, Ramona Schoedel, Florian Bemmann, Astrid Wirth, Christoph Sürig, Clemens Stachl, M. Bühner, Frank Niklas","doi":"10.1080/15305058.2022.2036160","DOIUrl":"https://doi.org/10.1080/15305058.2022.2036160","url":null,"abstract":"Abstract Digital technologies play an important role in our daily lives. Smartphones and tablet computers are very common worldwide and are available for everybody from a very early age. This trend offers the opportunity to track digital usage data for psychological and educational research purposes. The current paper introduces two research projects, the PhoneStudy and Learning4Kids that both use mobile sensing software to collect ecologically valid data on the usage of applications installed on smartphones and tablets. This usage data is used for statistical analyses, for a reward system, and to provide feedback to the study participants. The advantages and challenges of using mobile sensing compared to conventional forms of assessments, and the potential applications of mobile sensing in psychological and educational research are discussed.","PeriodicalId":46615,"journal":{"name":"International Journal of Testing","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2022-10-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43292336","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：