Pub Date : 2024-06-01Epub Date: 2024-02-01DOI: 10.1177/0193841X241227480
Burt S Barnow, Sanjay K Pandey, Qian Eric Luo
This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.
{"title":"How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations.","authors":"Burt S Barnow, Sanjay K Pandey, Qian Eric Luo","doi":"10.1177/0193841X241227480","DOIUrl":"10.1177/0193841X241227480","url":null,"abstract":"<p><p>This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"495-514"},"PeriodicalIF":0.9,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139651808","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-30DOI: 10.1177/0193841x241248864
Pamela R. Buckley, Katie Massey Combs, Karen M. Drewelow, Brittany L. Hubler, Marion Amanda Lain
As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers’ ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane’s (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.
{"title":"Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions","authors":"Pamela R. Buckley, Katie Massey Combs, Karen M. Drewelow, Brittany L. Hubler, Marion Amanda Lain","doi":"10.1177/0193841x241248864","DOIUrl":"https://doi.org/10.1177/0193841x241248864","url":null,"abstract":"As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers’ ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane’s (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"11 1","pages":""},"PeriodicalIF":0.9,"publicationDate":"2024-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140833906","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-16DOI: 10.1177/0193841x241246833
Bruno Arpino, Silvia Bacci, Leonardo Grilli, Raffaele Guetto, Carla Rampichini
We consider estimating the effect of a treatment on a given outcome measured on subjects tested both before and after treatment assignment in observational studies. A vast literature compares the competing approaches of modelling the post-test score conditionally on the pre-test score versus modelling the difference, namely, the gain score. Our contribution lies in analyzing the merits and drawbacks of two approaches in a multilevel setting. This is relevant in many fields, such as education, where students are nested within schools. The multilevel structure raises peculiar issues related to contextual effects and the distinction between individual-level and cluster-level treatments. We compare the two approaches through a simulation study. For individual-level treatments, our findings align with existing literature. However, for cluster-level treatments, the scenario is more complex, as the cluster mean of the pre-test score plays a key role. Its reliability crucially depends on the cluster size, leading to potentially unsatisfactory estimators with small clusters.
{"title":"Conditioning on the Pre-Test versus Gain Score Modelling: Revisiting the Controversy in a Multilevel Setting","authors":"Bruno Arpino, Silvia Bacci, Leonardo Grilli, Raffaele Guetto, Carla Rampichini","doi":"10.1177/0193841x241246833","DOIUrl":"https://doi.org/10.1177/0193841x241246833","url":null,"abstract":"We consider estimating the effect of a treatment on a given outcome measured on subjects tested both before and after treatment assignment in observational studies. A vast literature compares the competing approaches of modelling the post-test score conditionally on the pre-test score versus modelling the difference, namely, the gain score. Our contribution lies in analyzing the merits and drawbacks of two approaches in a multilevel setting. This is relevant in many fields, such as education, where students are nested within schools. The multilevel structure raises peculiar issues related to contextual effects and the distinction between individual-level and cluster-level treatments. We compare the two approaches through a simulation study. For individual-level treatments, our findings align with existing literature. However, for cluster-level treatments, the scenario is more complex, as the cluster mean of the pre-test score plays a key role. Its reliability crucially depends on the cluster size, leading to potentially unsatisfactory estimators with small clusters.","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"2012 1","pages":""},"PeriodicalIF":0.9,"publicationDate":"2024-04-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140612307","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-01Epub Date: 2023-06-06DOI: 10.1177/0193841X231181747
Mehmet Akif Destek, İbrahim Halil Oğuz, Nuh Okumuş
The adoption of growth strategies based on foreign trade, especially in the previous century when liberal policies began to dominate, is one of the main reasons for the increase in output and indirectly for environmental concerns. On the other hand, there are complex claims about the environmental effects of liberal policies and thus of globalization. This study intends to analyze the effects of global collaborations involving 11 transition economies that have completed the transition process on the environmentally sustainable development of these nations. In this direction, the effects of financial and commercial globalization indices on carbon emissions are investigated. The distinctions of globalization are used to distinguish the consequences of the two types of globalization. In doing so, the de facto and de jure indicator distinctions of globalization are used to differentiate the consequences of two types of globalization. In addition, the effects of real GDP, energy efficiency, and use of renewable energy on environmental pollution are dissected. For the main purpose of the study, the CS-ARDL estimation technique that allows cross-sectional dependency among observed countries is used to separate the short and long-run influences of explanatory variables. In addition, CCE-MG estimator is used for robustness check. According to the empirical findings, the economic growth and increasing energy intensity increases carbon emissions, but the increase in renewable energy consumption improves environmental quality. Furthermore, trade globalization does not have a significant impact on the environment in the context of globalization. On the other hand, the increase in de facto and de jure financial globalization indices results in an increase in carbon emissions, but de jure financial globalization causes more environmental damage. The harmful impact of de jure financial globalization on environmental quality suggests that the decreasing investment restrictions and international investment agreements of transition countries have been implemented in a manner that facilitates the relocation of investments from pollution-intensive industries to these countries.
{"title":"Do Trade and Financial Cooperation Improve Environmentally Sustainable Development: A Distinction Between <i>de facto</i> and <i>de jure</i> Globalization.","authors":"Mehmet Akif Destek, İbrahim Halil Oğuz, Nuh Okumuş","doi":"10.1177/0193841X231181747","DOIUrl":"10.1177/0193841X231181747","url":null,"abstract":"<p><p>The adoption of growth strategies based on foreign trade, especially in the previous century when liberal policies began to dominate, is one of the main reasons for the increase in output and indirectly for environmental concerns. On the other hand, there are complex claims about the environmental effects of liberal policies and thus of globalization. This study intends to analyze the effects of global collaborations involving 11 transition economies that have completed the transition process on the environmentally sustainable development of these nations. In this direction, the effects of financial and commercial globalization indices on carbon emissions are investigated. The distinctions of globalization are used to distinguish the consequences of the two types of globalization. In doing so, the de facto and de jure indicator distinctions of globalization are used to differentiate the consequences of two types of globalization. In addition, the effects of real GDP, energy efficiency, and use of renewable energy on environmental pollution are dissected. For the main purpose of the study, the CS-ARDL estimation technique that allows cross-sectional dependency among observed countries is used to separate the short and long-run influences of explanatory variables. In addition, CCE-MG estimator is used for robustness check. According to the empirical findings, the economic growth and increasing energy intensity increases carbon emissions, but the increase in renewable energy consumption improves environmental quality. Furthermore, trade globalization does not have a significant impact on the environment in the context of globalization. On the other hand, the increase in de facto and de jure financial globalization indices results in an increase in carbon emissions, but de jure financial globalization causes more environmental damage. The harmful impact of de jure financial globalization on environmental quality suggests that the decreasing investment restrictions and international investment agreements of transition countries have been implemented in a manner that facilitates the relocation of investments from pollution-intensive industries to these countries.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"251-273"},"PeriodicalIF":0.9,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9957122","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-01Epub Date: 2023-05-17DOI: 10.1177/0193841X231176869
Md Abul Kalam Azad, Mehedi Hasan Ifti, Chowdhury Noushin Novera, Anh Ngoc Quang Huynh, Esra Sipahi Döngül
The impact of pro-environmental behavior on policymaking has been an exciting area of research. While the relationship between pro-environmental behavior and policymaking has been explored in numerous studies, there needs to be more synthesis on this topic. This is the first text-mining study of pro-environmental effects in which policymaking is a significant factor. In response, this study, for the first time, takes a novel approach by using text mining in R programming to analyze 30 publications from the Scopus database on pro-environmental behavior in policymaking, highlighting major research themes and prospective research areas for future investigation. Results from text mining yielded 10 topic models, which are presented with a synopsis of the published research and a list of the primary authors, as well as a posterior probability via latent Dirichlet allocation (LDA). Additionally, the study conducts a trend analysis of the top 10 journals with the highest impact factor, considering the influence of each journal's mean citation. The study offers an overview of the impacts of pro-environmental behavior in policymaking, showing the most relevant and frequently discussed themes, introduces the scientific visualization of papers published in the Scopus database, and proposes future study directions. These findings can help researchers and environmental specialists better understand how pro-environmental behavior can be fostered more effectively through policymaking.
{"title":"Promoting Pro-Environmental Behavior in Policymaking: A Text-Mining Approach for Literature Review.","authors":"Md Abul Kalam Azad, Mehedi Hasan Ifti, Chowdhury Noushin Novera, Anh Ngoc Quang Huynh, Esra Sipahi Döngül","doi":"10.1177/0193841X231176869","DOIUrl":"10.1177/0193841X231176869","url":null,"abstract":"<p><p>The impact of pro-environmental behavior on policymaking has been an exciting area of research. While the relationship between pro-environmental behavior and policymaking has been explored in numerous studies, there needs to be more synthesis on this topic. This is the first text-mining study of pro-environmental effects in which policymaking is a significant factor. In response, this study, for the first time, takes a novel approach by using text mining in R programming to analyze 30 publications from the Scopus database on pro-environmental behavior in policymaking, highlighting major research themes and prospective research areas for future investigation. Results from text mining yielded 10 topic models, which are presented with a synopsis of the published research and a list of the primary authors, as well as a posterior probability via latent Dirichlet allocation (LDA). Additionally, the study conducts a trend analysis of the top 10 journals with the highest impact factor, considering the influence of each journal's mean citation. The study offers an overview of the impacts of pro-environmental behavior in policymaking, showing the most relevant and frequently discussed themes, introduces the scientific visualization of papers published in the Scopus database, and proposes future study directions. These findings can help researchers and environmental specialists better understand how pro-environmental behavior can be fostered more effectively through policymaking.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"370-398"},"PeriodicalIF":0.9,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9543356","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-01Epub Date: 2023-08-03DOI: 10.1177/0193841X231193468
Jin Zheng, Yao Xiong, Yimei Zheng, Haitao Zhang, Rui Wu
Stroke is the leading cause of death and disability among people in China, and it leads to heavy burdens for patients, their families and society. An accurate prediction of the risk of stroke has important implications for early intervention and treatment. In light of recent advances in machine learning, the application of this technique in stroke prediction has achieved plentiful promising results. To detect the relationship between potential factors and the risk of stroke and examine which machine learning method significantly can enhance the prediction accuracy of stroke. We employed six machine learning methods including logistic regression, naive Bayes, decision tree, random forest, K-nearest neighbor and support vector machine, to model and predict the risk of stroke. Participants were 233 patients from Sichuan and Chongqing. Four indicators (accuracy, precision, recall and F1 metric) were examined to evaluate the predictive performance of the different models. The empirical results indicate that random forest yields the best accuracy, recall and F1 in predicting the risk of stroke, with an accuracy of .7548, precision of .7805, recall of .7619 and F1 of .7711. Additionally, the findings show that age, cerebral infarction, PM 8 (an anti-atrial fibrillation drug), and drinking are independent risk factors for stroke. Further studies should adopt a broader assortment of machine learning methods to analyze the risk of stroke, by which better accuracy can be expected. In particular, RF can successfully enhance the forecasting accuracy for stroke.
脑卒中是导致中国人死亡和残疾的主要原因,给患者、家庭和社会带来沉重负担。准确预测脑卒中风险对早期干预和治疗具有重要意义。随着机器学习技术的不断进步,该技术在脑卒中预测中的应用也取得了丰硕的成果。为了检测潜在因素与脑卒中风险之间的关系,并研究哪种机器学习方法能显著提高脑卒中预测的准确性。我们采用了六种机器学习方法,包括逻辑回归、天真贝叶斯、决策树、随机森林、K-近邻和支持向量机,对脑卒中风险进行建模和预测。研究对象为来自四川和重庆的 233 名患者。研究考察了四个指标(准确度、精确度、召回率和 F1 指标),以评估不同模型的预测性能。实证结果表明,随机森林预测脑卒中风险的准确度、召回率和 F1 值最佳,准确度为 0.7548,精确度为 0.7805,召回率为 0.7619,F1 值为 0.7711。此外,研究结果表明,年龄、脑梗塞、PM 8(一种抗心房颤动药物)和饮酒是中风的独立危险因素。进一步的研究应采用更广泛的机器学习方法来分析中风风险,从而提高准确性。尤其是射频技术可以成功提高中风的预测准确性。
{"title":"Evaluating the Stroke Risk of Patients using Machine Learning: A New Perspective from Sichuan and Chongqing.","authors":"Jin Zheng, Yao Xiong, Yimei Zheng, Haitao Zhang, Rui Wu","doi":"10.1177/0193841X231193468","DOIUrl":"10.1177/0193841X231193468","url":null,"abstract":"<p><p>Stroke is the leading cause of death and disability among people in China, and it leads to heavy burdens for patients, their families and society. An accurate prediction of the risk of stroke has important implications for early intervention and treatment. In light of recent advances in machine learning, the application of this technique in stroke prediction has achieved plentiful promising results. To detect the relationship between potential factors and the risk of stroke and examine which machine learning method significantly can enhance the prediction accuracy of stroke. We employed six machine learning methods including logistic regression, naive Bayes, decision tree, random forest, K-nearest neighbor and support vector machine, to model and predict the risk of stroke. Participants were 233 patients from Sichuan and Chongqing. Four indicators (accuracy, precision, recall and F1 metric) were examined to evaluate the predictive performance of the different models. The empirical results indicate that random forest yields the best accuracy, recall and F1 in predicting the risk of stroke, with an accuracy of .7548, precision of .7805, recall of .7619 and F1 of .7711. Additionally, the findings show that age, cerebral infarction, PM 8 (an anti-atrial fibrillation drug), and drinking are independent risk factors for stroke. Further studies should adopt a broader assortment of machine learning methods to analyze the risk of stroke, by which better accuracy can be expected. In particular, RF can successfully enhance the forecasting accuracy for stroke.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"346-369"},"PeriodicalIF":0.9,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10302426","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-01Epub Date: 2023-06-12DOI: 10.1177/0193841X231175549
David R Judkins, Gabriel Durham
In 2003, Bloom, Hill, and Riccio (BHR) published an influential paper introducing novel methods for explaining the variation in local impacts observed in multi-site randomized control trials of socio-economic interventions in terms of site-level mediators. This paper seeks to improve upon this previous work by using student-level data to measure site-level mediators and confounders. Development of asymptotic behavior backed up with simulations and an empirical example. Students and training providers. Two simulations and an empirical application to data from an evaluation of the Health Professions Opportunity Grants (HPOG) Program. This empirical analysis involved roughly 6600 participants across 37 local sites. We examine bias and mean square error of estimates of mediation coefficients as well as the true coverage of nominal 95-percent confidence intervals on the mediation coefficients. Simulations suggest that the new methods generally improve the quality of inferences even when there is no confounding. Applying this methodology to the HPOG study shows that program-average FTE months of study by month six was a significant mediator of both career progress and long-term degree/credential receipt. Evaluators can robustify their BHR-style analyses by the use of the methods proposed here.
{"title":"Using Ecometric Data to Explore Sources of Cross-Site Impact Variance in Multi-Site Trials.","authors":"David R Judkins, Gabriel Durham","doi":"10.1177/0193841X231175549","DOIUrl":"10.1177/0193841X231175549","url":null,"abstract":"<p><p>In 2003, Bloom, Hill, and Riccio (BHR) published an influential paper introducing novel methods for explaining the variation in local impacts observed in multi-site randomized control trials of socio-economic interventions in terms of site-level mediators. This paper seeks to improve upon this previous work by using student-level data to measure site-level mediators and confounders. Development of asymptotic behavior backed up with simulations and an empirical example. Students and training providers. Two simulations and an empirical application to data from an evaluation of the Health Professions Opportunity Grants (HPOG) Program. This empirical analysis involved roughly 6600 participants across 37 local sites. We examine bias and mean square error of estimates of mediation coefficients as well as the true coverage of nominal 95-percent confidence intervals on the mediation coefficients. Simulations suggest that the new methods generally improve the quality of inferences even when there is no confounding. Applying this methodology to the HPOG study shows that program-average FTE months of study by month six was a significant mediator of both career progress and long-term degree/credential receipt. Evaluators can robustify their BHR-style analyses by the use of the methods proposed here.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"274-311"},"PeriodicalIF":0.9,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9969498","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-01Epub Date: 2023-05-08DOI: 10.1177/0193841X231171965
Serena Berretta, Sara Garbin, Maria Iannario, Omar Paccagnella
Program evaluations often investigate complex or multi-dimensional constructs, such as individual opinions or attitudes, by means of ratings. A different interpretation of the same question may affect cross-country comparability, leading to the Differential Item Functioning problem. Anchoring vignettes were introduced in the literature as a way to adjust self-evaluations from this interpersonal incomparability. In this paper, we first introduce a new nonparametric solution to analyse anchoring vignette data, recoding a variable based on a rating scale to a new corrected-variable that guarantees comparability in any cross-country analysis. Then, we exploit the flexibility of a mixture model introduced to account for uncertainty in the response process (the CUP model) to test if the proposed solution is effectively able to remove this reported heterogeneity. This solution is easy to construct and has important advantages compared with the original nonparametric solution adopted with anchoring vignette data. The novel indicator is applied to investigate self-reported depression in an old population. Data that will be analysed come from the second wave of the Survey of Health, Ageing and Retirement in Europe, collected in 2006/2007. Results highlight the need of correcting for reported heterogeneity comparing individual self-evaluations. Once interpersonal incomparability resulting from the different uses of response scales is removed from the self-assessments, some estimates are reversed in magnitude and signs with respect to the analysis of the collected data.
{"title":"A Novel Indicator to Correct for Individual Reported Heterogeneity. An Application to Self-Evaluation of Later-Life Depression.","authors":"Serena Berretta, Sara Garbin, Maria Iannario, Omar Paccagnella","doi":"10.1177/0193841X231171965","DOIUrl":"10.1177/0193841X231171965","url":null,"abstract":"<p><p>Program evaluations often investigate complex or multi-dimensional constructs, such as individual opinions or attitudes, by means of ratings. A different interpretation of the same question may affect cross-country comparability, leading to the Differential Item Functioning problem. Anchoring vignettes were introduced in the literature as a way to adjust self-evaluations from this interpersonal incomparability. In this paper, we first introduce a new nonparametric solution to analyse anchoring vignette data, recoding a variable based on a rating scale to a <i>new corrected-</i>variable that guarantees comparability in any cross-country analysis. Then, we exploit the flexibility of a mixture model introduced to account for uncertainty in the response process (the CUP model) to test if the proposed solution is effectively able to remove this reported heterogeneity. This solution is easy to construct and has important advantages compared with the original nonparametric solution adopted with anchoring vignette data. The novel indicator is applied to investigate self-reported depression in an old population. Data that will be analysed come from the second wave of the Survey of Health, Ageing and Retirement in Europe, collected in 2006/2007. Results highlight the need of correcting for reported heterogeneity comparing individual self-evaluations. Once interpersonal incomparability resulting from the different uses of response scales is removed from the self-assessments, some estimates are reversed in magnitude and signs with respect to the analysis of the collected data.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"221-250"},"PeriodicalIF":0.9,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9433536","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-01Epub Date: 2023-06-23DOI: 10.1177/0193841X231182749
Shiying Wang, Jaffar Abbas, Khalid Ibrahim Al-Sulati, Syed Ale Raza Shah
Economic corridors unlock new economic opportunities and tourism development in the region to achieve sustainable development goals. Green economic growth is conducive to environmental sustainability. Economic mega-projects of CPEC promote tourism that leads to communities' well-being and better quality of life. Modern infrastructure development contributes significantly to economic growth and tourism activities. This study's objectives emphasize exploring tourism and sustainable development pursuits under OBOR economic projects that open doors to improving residents' quality of life. The growing world is an eyewitness to a continuous rise in emissions and its severe consequences for humankind. It is necessary to show off the leading factors that result in tourism and economic activities causing environmental pollution rather than blame policymakers. Undoubtedly, many studies previously focused on demonstrating the influence of socio-economic factors that lead to better environmental quality. However, the empirical literature on tourism, social well-being, foreign direct investment, and the Environment in Belt and Road developed economies needed improvement. This research applied a series of advanced estimators that help demonstrate the study's probable results. This study explores the role of Social well-being (HDI), tourism development, FDI, renewable energy, information & communication technology (ICT), and urbanization on CO2 emissions in Belt and Road (BRI) developed economies.Estimated results exhibited the significant contribution of ICT and renewable energy to sustainability. Besides, FDI contributes to emissions reduction after its threshold level. Conversely, urbanization and tourism activities contribute to environmental pollution. The study outcomes stated inverted/EKC U-shaped hypotheses related to specified economies. Finally, the analysis based on the D-H panel causality test constructs exciting results.The present study concludes that economic corridor plays a vital role in tourism development, the community's well-being, and SDGs goals (sustainable development) impact on environmental safety. The findings suggest essential and applicable policies to attain the desired sustainability level. Findings contribute to the literature on tourism, well-being, and sustainability. Further studies can use insights using this methodology.
经济走廊为本地区带来新的经济机遇和旅游业发展,以实现可持续发展目标。绿色经济增长有利于环境的可持续发展。中巴经济走廊的大型经济项目促进了旅游业的发展,提高了社区的福祉和生活质量。现代基础设施发展极大地促进了经济增长和旅游活动。本研究的目标强调在 OBOR 经济项目下探索旅游业和可持续发展,为提高居民生活质量打开大门。不断发展的世界见证了排放量的持续增长及其对人类造成的严重后果。有必要揭示旅游和经济活动造成环境污染的主导因素,而不是指责政策制定者。毋庸置疑,以前的许多研究都侧重于展示社会经济因素对改善环境质量的影响。然而,有关 "一带一路 "发达经济体的旅游业、社会福利、外国直接投资和环境的实证文献有待改进。本研究采用了一系列先进的估计方法,有助于证明研究的可能结果。本研究探讨了 "一带一路 "发达经济体的社会福利(人类发展指数)、旅游业发展、外国直接投资、可再生能源、信息通信技术(ICT)和城市化对二氧化碳排放的影响。此外,外国直接投资在达到阈值后也有助于减排。相反,城市化和旅游活动加剧了环境污染。研究结果表明了与特定经济体相关的倒/EKC U 型假设。最后,基于 D-H 面板因果检验的分析得出了令人兴奋的结果。本研究得出结论,经济走廊在旅游业发展、社区福祉和可持续发展目标(可持续发展)对环境安全的影响方面发挥着至关重要的作用。研究结果提出了达到预期可持续发展水平的基本适用政策。研究结果为有关旅游业、福祉和可持续性的文献做出了贡献。进一步的研究可以利用这种方法获得更多的见解。
{"title":"The Impact of Economic Corridor and Tourism on Local Community's Quality of Life under One Belt One Road Context.","authors":"Shiying Wang, Jaffar Abbas, Khalid Ibrahim Al-Sulati, Syed Ale Raza Shah","doi":"10.1177/0193841X231182749","DOIUrl":"10.1177/0193841X231182749","url":null,"abstract":"<p><p>Economic corridors unlock new economic opportunities and tourism development in the region to achieve sustainable development goals. Green economic growth is conducive to environmental sustainability. Economic mega-projects of CPEC promote tourism that leads to communities' well-being and better quality of life. Modern infrastructure development contributes significantly to economic growth and tourism activities. This study's objectives emphasize exploring tourism and sustainable development pursuits under OBOR economic projects that open doors to improving residents' quality of life. The growing world is an eyewitness to a continuous rise in emissions and its severe consequences for humankind. It is necessary to show off the leading factors that result in tourism and economic activities causing environmental pollution rather than blame policymakers. Undoubtedly, many studies previously focused on demonstrating the influence of socio-economic factors that lead to better environmental quality. However, the empirical literature on tourism, social well-being, foreign direct investment, and the Environment in Belt and Road developed economies needed improvement. This research applied a series of advanced estimators that help demonstrate the study's probable results. This study explores the role of Social well-being (HDI), tourism development, FDI, renewable energy, information & communication technology (ICT), and urbanization on CO2 emissions in Belt and Road (BRI) developed economies.Estimated results exhibited the significant contribution of ICT and renewable energy to sustainability. Besides, FDI contributes to emissions reduction after its threshold level. Conversely, urbanization and tourism activities contribute to environmental pollution. The study outcomes stated inverted/EKC U-shaped hypotheses related to specified economies. Finally, the analysis based on the D-H panel causality test constructs exciting results.The present study concludes that economic corridor plays a vital role in tourism development, the community's well-being, and SDGs goals (sustainable development) impact on environmental safety. The findings suggest essential and applicable policies to attain the desired sustainability level. Findings contribute to the literature on tourism, well-being, and sustainability. Further studies can use insights using this methodology.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"312-345"},"PeriodicalIF":0.9,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10050838","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-03-04DOI: 10.1177/0193841x241238031
Douglas J. Besharov
In a 1987 article, Peter R. Rossi promulgated “The Iron Law of Evaluation and Other Metallic Rules.” The Metallic Laws were meant as an informal (and humorous) overstatement of the weakness of contemporary evaluations of social programs. Rossi’ s underlying worry was not so much about the state of evaluation technology in the abstract, but, rather, in its inability to advance our broad understanding of social problems and what to do about them---in other words, to make evaluation policy relevant. Rossi attributed the continuing failure to develop successful “large-scale social programs” to the failure to build a strong knowledge base for this kind of “social engineering.” The qualities of studies that enable such accumulated learning are variously labeled “external validity,” “generalizability,” “applicability,” or “transferability.” This Special Issue includes five papers that seek to explore and apply this understanding.
{"title":"Program Evaluation’s Path to Greater Policy Relevance: Learning From Rossi’s Iron Laws","authors":"Douglas J. Besharov","doi":"10.1177/0193841x241238031","DOIUrl":"https://doi.org/10.1177/0193841x241238031","url":null,"abstract":"In a 1987 article, Peter R. Rossi promulgated “The Iron Law of Evaluation and Other Metallic Rules.” The Metallic Laws were meant as an informal (and humorous) overstatement of the weakness of contemporary evaluations of social programs. Rossi’ s underlying worry was not so much about the state of evaluation technology in the abstract, but, rather, in its inability to advance our broad understanding of social problems and what to do about them---in other words, to make evaluation policy relevant. Rossi attributed the continuing failure to develop successful “large-scale social programs” to the failure to build a strong knowledge base for this kind of “social engineering.” The qualities of studies that enable such accumulated learning are variously labeled “external validity,” “generalizability,” “applicability,” or “transferability.” This Special Issue includes five papers that seek to explore and apply this understanding.","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"12 1","pages":""},"PeriodicalIF":0.9,"publicationDate":"2024-03-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140036990","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}