Pub Date : 2023-10-31DOI: 10.1177/13563890231205370
{"title":"French language abstracts","authors":"","doi":"10.1177/13563890231205370","DOIUrl":"https://doi.org/10.1177/13563890231205370","url":null,"abstract":"","PeriodicalId":47511,"journal":{"name":"Evaluation","volume":"115 2","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135872295","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-10-06DOI: 10.1177/13563890231204664
Tim Strasser, Joop de Kraker
Conventional evaluation and strategy approaches insufficiently address the needs of social innovation to adapt to non-linear and emergent change processes. This study addresses this shortcoming by testing a recently developed conceptual framework (3D) for the purpose of adaptive strategy and evaluation. We translated the 3D framework into a practice tool (SCALE 3D [Strategic Capacity development, Leadership and Evaluation in 3 Dimensions]) and applied it in two projects and four workshop settings through an action-research approach, involving networks of community-led sustainability initiatives. We describe practical benefits and suggest process steps for implementing SCALE 3D, as well as overall lessons learnt. We discuss how SCALE 3D can support transformation-oriented networks in alignment with adaptive strategy and evaluation approaches, to support strategic learning as well as reporting, and thereby help practitioners adapt to emerging changes and be accountable to funders. Our findings are relevant for evaluators, action researchers, strategy consultants, funders and social innovation practitioners supporting transformative networks.
{"title":"Applying SCALE 3D for evaluating transformative social innovation","authors":"Tim Strasser, Joop de Kraker","doi":"10.1177/13563890231204664","DOIUrl":"https://doi.org/10.1177/13563890231204664","url":null,"abstract":"Conventional evaluation and strategy approaches insufficiently address the needs of social innovation to adapt to non-linear and emergent change processes. This study addresses this shortcoming by testing a recently developed conceptual framework (3D) for the purpose of adaptive strategy and evaluation. We translated the 3D framework into a practice tool (SCALE 3D [Strategic Capacity development, Leadership and Evaluation in 3 Dimensions]) and applied it in two projects and four workshop settings through an action-research approach, involving networks of community-led sustainability initiatives. We describe practical benefits and suggest process steps for implementing SCALE 3D, as well as overall lessons learnt. We discuss how SCALE 3D can support transformation-oriented networks in alignment with adaptive strategy and evaluation approaches, to support strategic learning as well as reporting, and thereby help practitioners adapt to emerging changes and be accountable to funders. Our findings are relevant for evaluators, action researchers, strategy consultants, funders and social innovation practitioners supporting transformative networks.","PeriodicalId":47511,"journal":{"name":"Evaluation","volume":"213 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135350771","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-10-01DOI: 10.1177/13563890231204661
Daniel E. Esser, Heiner Janus
We analyse qualitative data collected from employees at Germany’s two main international development organisations, Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) and Kreditanstalt für Wiederaufbau (KfW) Development Bank, to study how upward accountability and organisational learning interact in the world’s second largest foreign aid system. Goffman’s ‘staging’ heuristic is applied to unpack social practices in these two organisations. We find that employees navigate two separate domains, a frontstage and a backstage. They consider the federal bureaucracy an audience expecting a coherent storyline despite the messy realities of foreign aid. In response, they engage in impression management on a frontstage while shielding their backstages from scrutiny to maximise autonomy. As a result, organisational learning at GIZ and KfW in Goffman’s terms focuses on collective efficacy at satisfying accountability expectations through staged performances. We relate these insights to the hierarchical structure of Germany’s foreign aid system, the role of organisational interests and prevailing professional norms.
{"title":"How are accountability and organisational learning related? A study of German bilateral development cooperation","authors":"Daniel E. Esser, Heiner Janus","doi":"10.1177/13563890231204661","DOIUrl":"https://doi.org/10.1177/13563890231204661","url":null,"abstract":"We analyse qualitative data collected from employees at Germany’s two main international development organisations, Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) and Kreditanstalt für Wiederaufbau (KfW) Development Bank, to study how upward accountability and organisational learning interact in the world’s second largest foreign aid system. Goffman’s ‘staging’ heuristic is applied to unpack social practices in these two organisations. We find that employees navigate two separate domains, a frontstage and a backstage. They consider the federal bureaucracy an audience expecting a coherent storyline despite the messy realities of foreign aid. In response, they engage in impression management on a frontstage while shielding their backstages from scrutiny to maximise autonomy. As a result, organisational learning at GIZ and KfW in Goffman’s terms focuses on collective efficacy at satisfying accountability expectations through staged performances. We relate these insights to the hierarchical structure of Germany’s foreign aid system, the role of organisational interests and prevailing professional norms.","PeriodicalId":47511,"journal":{"name":"Evaluation","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135607021","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-29DOI: 10.1177/13563890231200292
Seweryn Krupnik, Anna Szczucka, Monika Woźniak, Valérie Pattyn
Qualitative comparative analysis is gradually becoming more established in the evaluation field. The purpose of this article is to highlight the potential for evaluation research of engaging in consecutive rounds of this analysis. This is possible when approaching qualitative comparative analysis as a systematic strategy for configurational theorizing. To substantiate this potential, we present two evaluation studies on Research and Development subsidies for companies in Poland. Compared with the results of the first study, the findings of the subsequent consecutive qualitative comparative analysis studies were much more nuanced and helped in developing a full-fledged configurational program theory. In addition to elaborating on the strengths of a consecutive qualitative comparative analysis approach and the relevance of configurational program theories for evaluators, this article shares the main lessons learned in overcoming challenges common to such designs. Thus, concrete guidance is offered to researchers and evaluators who are willing to take configurational theorizing seriously.
{"title":"The potential of consecutive qualitative comparative analysis as a systematic strategy for configurational theorizing","authors":"Seweryn Krupnik, Anna Szczucka, Monika Woźniak, Valérie Pattyn","doi":"10.1177/13563890231200292","DOIUrl":"https://doi.org/10.1177/13563890231200292","url":null,"abstract":"Qualitative comparative analysis is gradually becoming more established in the evaluation field. The purpose of this article is to highlight the potential for evaluation research of engaging in consecutive rounds of this analysis. This is possible when approaching qualitative comparative analysis as a systematic strategy for configurational theorizing. To substantiate this potential, we present two evaluation studies on Research and Development subsidies for companies in Poland. Compared with the results of the first study, the findings of the subsequent consecutive qualitative comparative analysis studies were much more nuanced and helped in developing a full-fledged configurational program theory. In addition to elaborating on the strengths of a consecutive qualitative comparative analysis approach and the relevance of configurational program theories for evaluators, this article shares the main lessons learned in overcoming challenges common to such designs. Thus, concrete guidance is offered to researchers and evaluators who are willing to take configurational theorizing seriously.","PeriodicalId":47511,"journal":{"name":"Evaluation","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135243987","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-27DOI: 10.1177/13563890231201876
Gabriela Camacho Garland, Derek Beach
This article argues for the importance of theory and theorizing for an evaluation in the form of a process theory of change. A process theory of change centers its theoretical attention on key episodes that explain how things worked, in which the causal linkages are unpacked. The key lies in answering why actors do what they do (and thus whether these actions can be traced back to the intervention). This theorization has three steps: (1) definition of intervention and potential contribution; (2) theorization of potential contribution pathways; and (3) unpacking the process. This procedure is illustrated with a hypothetical example.
{"title":"Theorizing how interventions work in evaluation: Process-tracing methods and theorizing process theories of change","authors":"Gabriela Camacho Garland, Derek Beach","doi":"10.1177/13563890231201876","DOIUrl":"https://doi.org/10.1177/13563890231201876","url":null,"abstract":"This article argues for the importance of theory and theorizing for an evaluation in the form of a process theory of change. A process theory of change centers its theoretical attention on key episodes that explain how things worked, in which the causal linkages are unpacked. The key lies in answering why actors do what they do (and thus whether these actions can be traced back to the intervention). This theorization has three steps: (1) definition of intervention and potential contribution; (2) theorization of potential contribution pathways; and (3) unpacking the process. This procedure is illustrated with a hypothetical example.","PeriodicalId":47511,"journal":{"name":"Evaluation","volume":"61 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135580104","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-23DOI: 10.1177/13563890231200057
Bente van Oort, Hilda van ‘t Riet, Adriana Parejo Pagador, Rosana Lescrauwaet Noboa, Carolien Aantjes
While evaluations are critical for non-governmental organizations to strengthen their advocacy strategies, evaluators and advocates encounter many difficulties evaluating such efforts. This article discusses the contribution of the participatory process evaluation methodology to advocacy evaluation, using a Dutch global health advocacy program as a case study. As participatory process evaluation is a novel methodology in the field of advocacy, the article’s primary focus concerns the application and utility of the methodology. Findings suggest that participatory process evaluation in an advocacy context can provide insights into the implementation of advocacy tools and activities, encouraging reflection and leading to ideas and practical tools to strengthen advocacy efforts. While participatory process evaluation can help overcome some of the often-experienced barriers in advocacy evaluation, further research is needed to consolidate advocacy evaluation theory and practice.
{"title":"Understanding the what, how, and why in advocacy: Assessing the applicability of participatory process evaluation methodology in an advocacy context","authors":"Bente van Oort, Hilda van ‘t Riet, Adriana Parejo Pagador, Rosana Lescrauwaet Noboa, Carolien Aantjes","doi":"10.1177/13563890231200057","DOIUrl":"https://doi.org/10.1177/13563890231200057","url":null,"abstract":"While evaluations are critical for non-governmental organizations to strengthen their advocacy strategies, evaluators and advocates encounter many difficulties evaluating such efforts. This article discusses the contribution of the participatory process evaluation methodology to advocacy evaluation, using a Dutch global health advocacy program as a case study. As participatory process evaluation is a novel methodology in the field of advocacy, the article’s primary focus concerns the application and utility of the methodology. Findings suggest that participatory process evaluation in an advocacy context can provide insights into the implementation of advocacy tools and activities, encouraging reflection and leading to ideas and practical tools to strengthen advocacy efforts. While participatory process evaluation can help overcome some of the often-experienced barriers in advocacy evaluation, further research is needed to consolidate advocacy evaluation theory and practice.","PeriodicalId":47511,"journal":{"name":"Evaluation","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135959709","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-23DOI: 10.1177/13563890231200291
Steffen Bohni Nielsen, Sofie Østergaard Jaspers, Sebastian Lemire
Realist evaluation and experimental designs are both well-established approaches to evaluation. Over the past 10 years, realist trials—evaluations purposefully combining realist evaluation and experimental designs—have emerged. Informed by a comprehensive review of published realist trials, this article examines to what extent and how realist trials align with quality standards for realist evaluations and randomized controlled trials and to what extent and how the realist and trial aspects of realist trials are integrated. We identified only few examples that met high-quality standards for both experimental and realist studies and that merged the two designs.
{"title":"The curious case of the realist trial: Methodological oxymoron or unicorn?","authors":"Steffen Bohni Nielsen, Sofie Østergaard Jaspers, Sebastian Lemire","doi":"10.1177/13563890231200291","DOIUrl":"https://doi.org/10.1177/13563890231200291","url":null,"abstract":"Realist evaluation and experimental designs are both well-established approaches to evaluation. Over the past 10 years, realist trials—evaluations purposefully combining realist evaluation and experimental designs—have emerged. Informed by a comprehensive review of published realist trials, this article examines to what extent and how realist trials align with quality standards for realist evaluations and randomized controlled trials and to what extent and how the realist and trial aspects of realist trials are integrated. We identified only few examples that met high-quality standards for both experimental and realist studies and that merged the two designs.","PeriodicalId":47511,"journal":{"name":"Evaluation","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135959880","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-14DOI: 10.1177/13563890231196601
Steve Powell, James Copestake, Fiona Remnant
Evaluators are interested in capturing how things causally influence one another. They are also interested in capturing how stakeholders think things causally influence one another. Causal mapping – the collection, coding and visualisation of interconnected causal claims – has been used widely for several decades across many disciplines for this purpose. It makes the provenance or source of such claims explicit and provides tools for gathering and dealing with this kind of data and for managing its Janus-like double-life: on the one hand, providing information about what people believe causes what, and on the other hand, preparing this information for possible evaluative judgements about what causes what. Specific reference to causal mapping in the evaluation literature is sparse, which we aim to redress here. In particular, the authors address the Janus dilemma by suggesting that causal maps can be understood neither as models of beliefs about causal pathways nor as models of causal pathways per se but as repositories of evidence for those pathways.
{"title":"Causal mapping for evaluators","authors":"Steve Powell, James Copestake, Fiona Remnant","doi":"10.1177/13563890231196601","DOIUrl":"https://doi.org/10.1177/13563890231196601","url":null,"abstract":"Evaluators are interested in capturing how things causally influence one another. They are also interested in capturing how stakeholders think things causally influence one another. Causal mapping – the collection, coding and visualisation of interconnected causal claims – has been used widely for several decades across many disciplines for this purpose. It makes the provenance or source of such claims explicit and provides tools for gathering and dealing with this kind of data and for managing its Janus-like double-life: on the one hand, providing information about what people believe causes what, and on the other hand, preparing this information for possible evaluative judgements about what causes what. Specific reference to causal mapping in the evaluation literature is sparse, which we aim to redress here. In particular, the authors address the Janus dilemma by suggesting that causal maps can be understood neither as models of beliefs about causal pathways nor as models of causal pathways per se but as repositories of evidence for those pathways.","PeriodicalId":47511,"journal":{"name":"Evaluation","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134911748","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-14DOI: 10.1177/13563890231196603
John Guenther, Ian Falk, Michael J. Cole
This article argues that not only is theory building from qualitative evaluation possible, but ought to be considered as a desirable product and utilisation of evaluators’ work. Based on three case studies, the authors show how theory building can work, and why it can be important and useful. Theory building is seldom considered an impact-producing product from evaluation. Theory in evaluation is typically limited to ‘evaluation theory’ as a way of explaining why and how different approaches to evaluation work. Theory is also used to inform programme design and ‘theory of change’. The literature seldom suggests that evaluations can be used to build theory in social sciences. The argument presented in this article builds on literature of ‘theories of evaluation use’ to suggest that theorising is a form of knowledge utilisation arising from well-constructed, open-ended evaluation questions, and conceptual use of findings and recommendations.
{"title":"Theory building from qualitative evaluation","authors":"John Guenther, Ian Falk, Michael J. Cole","doi":"10.1177/13563890231196603","DOIUrl":"https://doi.org/10.1177/13563890231196603","url":null,"abstract":"This article argues that not only is theory building from qualitative evaluation possible, but ought to be considered as a desirable product and utilisation of evaluators’ work. Based on three case studies, the authors show how theory building can work, and why it can be important and useful. Theory building is seldom considered an impact-producing product from evaluation. Theory in evaluation is typically limited to ‘evaluation theory’ as a way of explaining why and how different approaches to evaluation work. Theory is also used to inform programme design and ‘theory of change’. The literature seldom suggests that evaluations can be used to build theory in social sciences. The argument presented in this article builds on literature of ‘theories of evaluation use’ to suggest that theorising is a form of knowledge utilisation arising from well-constructed, open-ended evaluation questions, and conceptual use of findings and recommendations.","PeriodicalId":47511,"journal":{"name":"Evaluation","volume":"75 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134911733","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-09-14DOI: 10.1177/13563890231195906
Daria-Maria Gerke, Katrin Uude, Thorsten Kliewe
Academia focuses on the interplay of Higher Education Institutions and external stakeholders. In this context, academia is concerned with the societal impact and impact created in interactions with external stakeholders; the latter is often referred to as impact co-creation. There is agreement that the related processes leading to an impact are complex and multi-dimensional. However, academics disagree on how the ultimate, wider impact of research should be measured. This study seeks to conceptualize societal impact through the lens of value co-creation, arguing that societal impact is best conceptualized as the uptake of research. Based on this, we developed a generic research impact assessment framework to facilitate evaluations and enable cross-sector learning. This study contributes to academia by providing an overarching understanding of impact creation, including wider research impact, and offers the perspective that any research project involving stakeholders to a certain extent, also entails co-production.
{"title":"Co-creation and societal impact: Toward a generic framework for research impact assessment","authors":"Daria-Maria Gerke, Katrin Uude, Thorsten Kliewe","doi":"10.1177/13563890231195906","DOIUrl":"https://doi.org/10.1177/13563890231195906","url":null,"abstract":"Academia focuses on the interplay of Higher Education Institutions and external stakeholders. In this context, academia is concerned with the societal impact and impact created in interactions with external stakeholders; the latter is often referred to as impact co-creation. There is agreement that the related processes leading to an impact are complex and multi-dimensional. However, academics disagree on how the ultimate, wider impact of research should be measured. This study seeks to conceptualize societal impact through the lens of value co-creation, arguing that societal impact is best conceptualized as the uptake of research. Based on this, we developed a generic research impact assessment framework to facilitate evaluations and enable cross-sector learning. This study contributes to academia by providing an overarching understanding of impact creation, including wider research impact, and offers the perspective that any research project involving stakeholders to a certain extent, also entails co-production.","PeriodicalId":47511,"journal":{"name":"Evaluation","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134912555","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：