Pub Date : 2023-06-19DOI: 10.1080/10627197.2023.2222588
S. Finney, M. McFadden
ABSTRACT Questioning individuals about future behavior influences the subsequent performance of that behavior. This question-behavior effect may be enhanced when questions include positive self-identity prompts. To evaluate the effectiveness of this intervention in a testing context, we randomly assigned examinees to one of three conditions prior to completing a low-stakes test: answering five questions regarding intended effort prior to the test, answering five questions regarding intended effort that referenced positive self-identity prior to the test, or a control condition. We then administered two multiple-choice tests and collected two measures of test-taking effort: self-reported effort and response time effort. Answering questions about effort prior to completing the tests resulted in higher self-reported effort and perceived test importance. Moreover, in the question conditions, significantly fewer examinees were identified to be filtered from the dataset due to low effort. The easy strategy of asking examinees to report their intended effort appears to increase test-taking effort.
{"title":"Examining the Question-Behavior Effect in Low-Stakes Testing Contexts: A Cheap Strategy to Increase Examinee Effort","authors":"S. Finney, M. McFadden","doi":"10.1080/10627197.2023.2222588","DOIUrl":"https://doi.org/10.1080/10627197.2023.2222588","url":null,"abstract":"ABSTRACT Questioning individuals about future behavior influences the subsequent performance of that behavior. This question-behavior effect may be enhanced when questions include positive self-identity prompts. To evaluate the effectiveness of this intervention in a testing context, we randomly assigned examinees to one of three conditions prior to completing a low-stakes test: answering five questions regarding intended effort prior to the test, answering five questions regarding intended effort that referenced positive self-identity prior to the test, or a control condition. We then administered two multiple-choice tests and collected two measures of test-taking effort: self-reported effort and response time effort. Answering questions about effort prior to completing the tests resulted in higher self-reported effort and perceived test importance. Moreover, in the question conditions, significantly fewer examinees were identified to be filtered from the dataset due to low effort. The easy strategy of asking examinees to report their intended effort appears to increase test-taking effort.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":" ","pages":""},"PeriodicalIF":1.5,"publicationDate":"2023-06-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46037789","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-06-16DOI: 10.1080/10627197.2023.2222585
Maoxin Zhang, Björn Andersson
ABSTRACT Process data from educational assessments enhance the understanding of how students answer cognitive items. However, effectively making use of these data is challenging. We propose an approach to identify solution patterns from operation sequences and response times by generating networks from process data and defining network features that extract essential information from them. With these features, we group respondents to a problem-solving task from PISA 2012 using Gaussian mixture models. The results indicate the presence of two and four clusters for groups defined by failure and success on the task, respectively. We interpret the clusters as less-able, low-effort, adaptable, back-and-forth, deliberate, and trial-and-error clusters by considering the cluster-specific feature statistics. The proposed approach sheds light on students’ problem-solving mental processes, which can aid item development and facilitate individualized feedback to students. The method is applicable to many computer-based problems, but a limitation is that the feature definitions can be task-dependent.
{"title":"Identifying Problem-Solving Solution Patterns Using Network Analysis of Operation Sequences and Response Times","authors":"Maoxin Zhang, Björn Andersson","doi":"10.1080/10627197.2023.2222585","DOIUrl":"https://doi.org/10.1080/10627197.2023.2222585","url":null,"abstract":"ABSTRACT Process data from educational assessments enhance the understanding of how students answer cognitive items. However, effectively making use of these data is challenging. We propose an approach to identify solution patterns from operation sequences and response times by generating networks from process data and defining network features that extract essential information from them. With these features, we group respondents to a problem-solving task from PISA 2012 using Gaussian mixture models. The results indicate the presence of two and four clusters for groups defined by failure and success on the task, respectively. We interpret the clusters as less-able, low-effort, adaptable, back-and-forth, deliberate, and trial-and-error clusters by considering the cluster-specific feature statistics. The proposed approach sheds light on students’ problem-solving mental processes, which can aid item development and facilitate individualized feedback to students. The method is applicable to many computer-based problems, but a limitation is that the feature definitions can be task-dependent.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":"28 1","pages":"172 - 189"},"PeriodicalIF":1.5,"publicationDate":"2023-06-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41526853","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-05-21DOI: 10.1080/10627197.2023.2215977
Jihyun Lee, Jung‐Sook Lee, N. Ellis
ABSTRACT This study examines public opinion about the National Assessment Program – Literacy and Numeracy (NAPLAN) in Australia, based on an online survey conducted with a sample of New South Wales (NSW) residents (n = 2,017). Our study participants did recognize the benefits of NAPLAN despite their opinion generally trending toward negative viewpoints of the testing program. The views about the positive aspects of NAPLAN varied across socio-demographic groups, but there were more similar views concerning the negative views of NAPLAN. As predicted by the social-cultural and institutional performance models, those living in metropolitan areas and those from high SES groups tended to possess stronger positive views. Political party preference was a strong predictor of the perceptions about NAPLAN. Overall, this study offers possible explanations for the underlying mechanisms explaining sub-group differences in attitudes toward large-scale standardized national testing.
{"title":"Public Opinion About National Large-Scale Student Assessment: A Case of NAPLAN","authors":"Jihyun Lee, Jung‐Sook Lee, N. Ellis","doi":"10.1080/10627197.2023.2215977","DOIUrl":"https://doi.org/10.1080/10627197.2023.2215977","url":null,"abstract":"ABSTRACT This study examines public opinion about the National Assessment Program – Literacy and Numeracy (NAPLAN) in Australia, based on an online survey conducted with a sample of New South Wales (NSW) residents (n = 2,017). Our study participants did recognize the benefits of NAPLAN despite their opinion generally trending toward negative viewpoints of the testing program. The views about the positive aspects of NAPLAN varied across socio-demographic groups, but there were more similar views concerning the negative views of NAPLAN. As predicted by the social-cultural and institutional performance models, those living in metropolitan areas and those from high SES groups tended to possess stronger positive views. Political party preference was a strong predictor of the perceptions about NAPLAN. Overall, this study offers possible explanations for the underlying mechanisms explaining sub-group differences in attitudes toward large-scale standardized national testing.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":"28 1","pages":"137 - 155"},"PeriodicalIF":1.5,"publicationDate":"2023-05-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41772940","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-05-18DOI: 10.1080/10627197.2023.2213432
J. Soland, Megan Kuhfeld, Brennan Register
ABSTRACT Much of what we know about how children develop is based on survey data. In order to estimate growth across time and, thereby, better understand that development, short survey scales are typically administered at repeated timepoints. Before estimating growth, those repeated measures must be put onto the same scale. Yet, little research examines how scaling decisions affect comparisons of growth derived from survey item responses. In this study, we use a sample of 174,669 students in grades 7 through 12 who took the same self-efficacy and social awareness surveys for four years. We use those survey item responses to construct scales using different approaches, then compare the resultant scores to see how inferences about changes over time during adolescence might shift dependent on scaling. While we find that conclusions about average trends are quite consistent by scaling approach, specific quantifications of change like effect sizes can differ by scaling method.
{"title":"A Comparison of Methodologies for Scaling Longitudinal Social-Emotional Survey Responses","authors":"J. Soland, Megan Kuhfeld, Brennan Register","doi":"10.1080/10627197.2023.2213432","DOIUrl":"https://doi.org/10.1080/10627197.2023.2213432","url":null,"abstract":"ABSTRACT Much of what we know about how children develop is based on survey data. In order to estimate growth across time and, thereby, better understand that development, short survey scales are typically administered at repeated timepoints. Before estimating growth, those repeated measures must be put onto the same scale. Yet, little research examines how scaling decisions affect comparisons of growth derived from survey item responses. In this study, we use a sample of 174,669 students in grades 7 through 12 who took the same self-efficacy and social awareness surveys for four years. We use those survey item responses to construct scales using different approaches, then compare the resultant scores to see how inferences about changes over time during adolescence might shift dependent on scaling. While we find that conclusions about average trends are quite consistent by scaling approach, specific quantifications of change like effect sizes can differ by scaling method.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":"28 1","pages":"156 - 171"},"PeriodicalIF":1.5,"publicationDate":"2023-05-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46872564","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-04-17DOI: 10.1080/10627197.2023.2202311
Utkun Aydın, B. Birgili
ABSTRACT Internationally, mathematics education reform has been directed toward characterizing educational goals that go beyond topic/content/skill descriptions and develop students’ problem solving. The Revised Bloom’s Taxonomy and MATH (Mathematical Assessment Task Hierarchy) Taxonomy characterize such goals. University entrance examinations have been seen as one way of accomplishing these goals and influence learning, teaching, and assessment in mathematics. The present study analyzed mathematics items (N = 1077) in Turkish university entrance examinations in 1998-2013 and objectives (N = 621) in mathematics curricula in 2005, 2011, and 2013 to determine the extent to which they represent the dimensions/categories of these taxonomies and the degree to which items are aligned with objectives in terms of reflecting the dimensions/categories of these taxonomies. The findings reveal that the items demand, to a large extent, automated computational skills; this is also evident in the relevant mathematics curricula. Implications for practice are discussed and could play a role in reforming assessment.
{"title":"Assessing Mathematical Higher-Order Thinking Skills: An Analysis of Turkish University Entrance Examinations","authors":"Utkun Aydın, B. Birgili","doi":"10.1080/10627197.2023.2202311","DOIUrl":"https://doi.org/10.1080/10627197.2023.2202311","url":null,"abstract":"ABSTRACT Internationally, mathematics education reform has been directed toward characterizing educational goals that go beyond topic/content/skill descriptions and develop students’ problem solving. The Revised Bloom’s Taxonomy and MATH (Mathematical Assessment Task Hierarchy) Taxonomy characterize such goals. University entrance examinations have been seen as one way of accomplishing these goals and influence learning, teaching, and assessment in mathematics. The present study analyzed mathematics items (N = 1077) in Turkish university entrance examinations in 1998-2013 and objectives (N = 621) in mathematics curricula in 2005, 2011, and 2013 to determine the extent to which they represent the dimensions/categories of these taxonomies and the degree to which items are aligned with objectives in terms of reflecting the dimensions/categories of these taxonomies. The findings reveal that the items demand, to a large extent, automated computational skills; this is also evident in the relevant mathematics curricula. Implications for practice are discussed and could play a role in reforming assessment.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":"28 1","pages":"190 - 209"},"PeriodicalIF":1.5,"publicationDate":"2023-04-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48128926","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-04-03DOI: 10.1080/10627197.2023.2223924
Jennifer Randall
ABSTRACT In a justice-oriented antiracist assessment process, attention to the disruption of white supremacy must occur at every stage – from construct articulation to score reporting. An important step in the assessment development process is the item review stage often referred to as Bias/Fairness and Sensitivity Review. I argue that typical approaches to the item and test review process miss the opportunity to actively disrupt white supremacist and racist logics – in other words, to be anti-racist. Using Critical Race and Critical Whiteness Theory as a frame, this paper challenges the field to re-envision the purpose and outcomes of the bias and sensitivity review process by (a) identifying common themes and/or recommendations found in bias and sensitivity guidelines that, even if unintentionally, center whiteness and/or the paradigm of white dominant culture; and (b) recommending a set of bias and sensitivity principles that promote an antiracist approach to assessment design, specifically item review.
{"title":"It Ain’t Near ‘Bout Fair: Re-Envisioning the Bias and Sensitivity Review Process from a Justice-Oriented Antiracist Perspective","authors":"Jennifer Randall","doi":"10.1080/10627197.2023.2223924","DOIUrl":"https://doi.org/10.1080/10627197.2023.2223924","url":null,"abstract":"ABSTRACT In a justice-oriented antiracist assessment process, attention to the disruption of white supremacy must occur at every stage – from construct articulation to score reporting. An important step in the assessment development process is the item review stage often referred to as Bias/Fairness and Sensitivity Review. I argue that typical approaches to the item and test review process miss the opportunity to actively disrupt white supremacist and racist logics – in other words, to be anti-racist. Using Critical Race and Critical Whiteness Theory as a frame, this paper challenges the field to re-envision the purpose and outcomes of the bias and sensitivity review process by (a) identifying common themes and/or recommendations found in bias and sensitivity guidelines that, even if unintentionally, center whiteness and/or the paradigm of white dominant culture; and (b) recommending a set of bias and sensitivity principles that promote an antiracist approach to assessment design, specifically item review.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":"28 1","pages":"68 - 82"},"PeriodicalIF":1.5,"publicationDate":"2023-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48704460","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-04-03DOI: 10.1080/10627197.2023.2215980
J. Herman, J. Martínez, A. Bailey
ABSTRACT This special issue of Educational Assessment seeks to encourage reflection and discussion around the different assumptions and conceptualizations of fairness in assessment and their potential links to and implications for the next edition of the Standards for Educational and Psychological Testing. In this final commentary, the special issue editors summarize the major points advanced by the three contributing authors, and consider the variety of conceptual, methodological, and practical challenges and questions raised. We discuss a range of remaining issues requiring additional theorizing and empirical research to further illuminate and bring the ideas of the contributors to fruition. Finally, we highlight areas with direct implications to be considered in the development of the next edition of the Standards.
{"title":"Fairness in Educational Assessment and the Next Edition of the Standards: Concluding Commentary","authors":"J. Herman, J. Martínez, A. Bailey","doi":"10.1080/10627197.2023.2215980","DOIUrl":"https://doi.org/10.1080/10627197.2023.2215980","url":null,"abstract":"ABSTRACT This special issue of Educational Assessment seeks to encourage reflection and discussion around the different assumptions and conceptualizations of fairness in assessment and their potential links to and implications for the next edition of the Standards for Educational and Psychological Testing. In this final commentary, the special issue editors summarize the major points advanced by the three contributing authors, and consider the variety of conceptual, methodological, and practical challenges and questions raised. We discuss a range of remaining issues requiring additional theorizing and empirical research to further illuminate and bring the ideas of the contributors to fruition. Finally, we highlight areas with direct implications to be considered in the development of the next edition of the Standards.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":"28 1","pages":"128 - 136"},"PeriodicalIF":1.5,"publicationDate":"2023-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42510549","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-04-03DOI: 10.1080/10627197.2023.2215978
R. Bennett
ABSTRACT “Toward a Theory of Socioculturally Responsive Assessment” assembled design principles from multiple literatures and wove them into a working definition and a network of empirically testable propositions. The intention was to offer a coherent theoretical framework within which to understand why and how particular assessment designs might work, what actions testing programs should consider, how they might move forward with those actions, and how to evaluate the impact. Dr. Solano Flores offers many comments on these ideas, with which I mostly agree. In this response, I detail those agreements, as well as some points of departure. I close with some implications for revising the Standards.
{"title":"Let’s Agree to (Mostly) Agree: A Response to Solano-Flores","authors":"R. Bennett","doi":"10.1080/10627197.2023.2215978","DOIUrl":"https://doi.org/10.1080/10627197.2023.2215978","url":null,"abstract":"ABSTRACT “Toward a Theory of Socioculturally Responsive Assessment” assembled design principles from multiple literatures and wove them into a working definition and a network of empirically testable propositions. The intention was to offer a coherent theoretical framework within which to understand why and how particular assessment designs might work, what actions testing programs should consider, how they might move forward with those actions, and how to evaluate the impact. Dr. Solano Flores offers many comments on these ideas, with which I mostly agree. In this response, I detail those agreements, as well as some points of departure. I close with some implications for revising the Standards.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":"28 1","pages":"122 - 127"},"PeriodicalIF":1.5,"publicationDate":"2023-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47790677","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-04-03DOI: 10.1080/10627197.2023.2215979
J. Herman, A. Bailey, J. Martínez
ABSTRACT This introduction provides context for Educational Assessment’s special issue, ”Fairness in Educational Assessment and the Next Edition of the Standards.” The article introduces the topic of fairness by citing a prior Special Issue on which the current issue is built, summarizes the current Fairness Standards of the Standards for Educational and Psychological Testing (2014) and provides an overview to the issue. The issue includes focal articles by Dr. Jennifer Randall and Dr. Randy Bennett and a synthesis discussion by Dr. Guillermo Solano-Flores. The two focal authors then respond to Dr. Solano-Flores and the special issue editors end the issue with a concluding commentary.
{"title":"Introduction to the Special Issue: Fairness in Educational Assessment and the Next Edition of the Standards","authors":"J. Herman, A. Bailey, J. Martínez","doi":"10.1080/10627197.2023.2215979","DOIUrl":"https://doi.org/10.1080/10627197.2023.2215979","url":null,"abstract":"ABSTRACT This introduction provides context for Educational Assessment’s special issue, ”Fairness in Educational Assessment and the Next Edition of the Standards.” The article introduces the topic of fairness by citing a prior Special Issue on which the current issue is built, summarizes the current Fairness Standards of the Standards for Educational and Psychological Testing (2014) and provides an overview to the issue. The issue includes focal articles by Dr. Jennifer Randall and Dr. Randy Bennett and a synthesis discussion by Dr. Guillermo Solano-Flores. The two focal authors then respond to Dr. Solano-Flores and the special issue editors end the issue with a concluding commentary.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":"28 1","pages":"65 - 67"},"PeriodicalIF":1.5,"publicationDate":"2023-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49230336","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-04-03DOI: 10.1080/10627197.2023.2202312
R. Bennett
ABSTRACT In the United States, opposition to traditional standardized tests is widespread, particularly obvious in the admissions context but also evident in elementary and secondary education. This opposition is fueled in significant part by the perception that tests perpetuate social injustice through their content, design, and use. To survive, as well as contribute positively, the measurement field must rethink assessment, including how to make it more socioculturally responsive. This paper offers a rationale for that rethinking and then employs provisional design principles drawn from various literatures to formulate a working definition and the beginnings of a theory. In the closing section, a path toward implementation is suggested.
{"title":"Toward a Theory of Socioculturally Responsive Assessment","authors":"R. Bennett","doi":"10.1080/10627197.2023.2202312","DOIUrl":"https://doi.org/10.1080/10627197.2023.2202312","url":null,"abstract":"ABSTRACT In the United States, opposition to traditional standardized tests is widespread, particularly obvious in the admissions context but also evident in elementary and secondary education. This opposition is fueled in significant part by the perception that tests perpetuate social injustice through their content, design, and use. To survive, as well as contribute positively, the measurement field must rethink assessment, including how to make it more socioculturally responsive. This paper offers a rationale for that rethinking and then employs provisional design principles drawn from various literatures to formulate a working definition and the beginnings of a theory. In the closing section, a path toward implementation is suggested.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":"28 1","pages":"83 - 104"},"PeriodicalIF":1.5,"publicationDate":"2023-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43214871","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：