Leslie A. Fierro, Isabelle Bourgeois, R. Gokiert, Michelle Searle, Melissa Tremblay
Despite the attention evaluation capacity building (ECB) has received over the past several decades, surprisingly few opportunities for learning about ECB exist. In response to this need, the Max Bell School of Public Policy at McGill University, in collaboration with ECB scholars across Canada, created a case competition focused exclusively on ECB—the Evaluation Capacity Case Challenge (EC3). Twenty individuals interested in learning about ECB and one organization (case site) interested in enhancing their existing evaluation capacity were selected to participate through a competitive application process. Participants attended a series of online workshops and engaged with an ECB coach to hone their skills and then took part in a two-day hybrid case challenge event where they had 24 hours to craft an ECB plan in response to a specific case challenge question presented by case site representatives. In this article, the authors describe EC3 in detail and share some key reflections from the inaugural year.
{"title":"Strengthening Evaluation Capacity Building Practice Through Competition: The Max Bell School of Public Policy’s Evaluation Capacity Case Challenge","authors":"Leslie A. Fierro, Isabelle Bourgeois, R. Gokiert, Michelle Searle, Melissa Tremblay","doi":"10.3138/cjpe-2024-0012","DOIUrl":"https://doi.org/10.3138/cjpe-2024-0012","url":null,"abstract":"Despite the attention evaluation capacity building (ECB) has received over the past several decades, surprisingly few opportunities for learning about ECB exist. In response to this need, the Max Bell School of Public Policy at McGill University, in collaboration with ECB scholars across Canada, created a case competition focused exclusively on ECB—the Evaluation Capacity Case Challenge (EC3). Twenty individuals interested in learning about ECB and one organization (case site) interested in enhancing their existing evaluation capacity were selected to participate through a competitive application process. Participants attended a series of online workshops and engaged with an ECB coach to hone their skills and then took part in a two-day hybrid case challenge event where they had 24 hours to craft an ECB plan in response to a specific case challenge question presented by case site representatives. In this article, the authors describe EC3 in detail and share some key reflections from the inaugural year.","PeriodicalId":504262,"journal":{"name":"Canadian Journal of Program Evaluation","volume":"29 28","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140752942","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluation capacity building (ECB) is a well-established domain of scholarship and practice within the broader field of evaluation. Stemming from a concern to ensure sustainable and useful evaluation practices in organizations, ECB draws on several evaluation theories and approaches such as collaborative evaluation and utilization-focused evaluation. In a recent integrative review, the authors and their colleagues identified other theories, frameworks, and models that inform ECB; many of these were developed in other fields, such as social sciences and public administration. The present exploratory study sought to identify some of the knowledge and theories that inform ECB from other fields through a review of 47 “ECB-adjacent” articles. This article describes the various themes and topics included in these articles to formulate potential future avenues for ECB research and practice.
{"title":"Exploring the Edges: Identifying the Next Generation of Evaluation Capacity Building Research and Practice Through Adjacency","authors":"Leslie A. Fierro, Isabelle Bourgeois","doi":"10.3138/cjpe-2024-0014","DOIUrl":"https://doi.org/10.3138/cjpe-2024-0014","url":null,"abstract":"Evaluation capacity building (ECB) is a well-established domain of scholarship and practice within the broader field of evaluation. Stemming from a concern to ensure sustainable and useful evaluation practices in organizations, ECB draws on several evaluation theories and approaches such as collaborative evaluation and utilization-focused evaluation. In a recent integrative review, the authors and their colleagues identified other theories, frameworks, and models that inform ECB; many of these were developed in other fields, such as social sciences and public administration. The present exploratory study sought to identify some of the knowledge and theories that inform ECB from other fields through a review of 47 “ECB-adjacent” articles. This article describes the various themes and topics included in these articles to formulate potential future avenues for ECB research and practice.","PeriodicalId":504262,"journal":{"name":"Canadian Journal of Program Evaluation","volume":"126 ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140751755","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In 2023, McGill University’s Max Bell School of Public Policy hosted the inaugural Evaluation Capacity Case Challenge (EC3) competition with a cohort of 19 selected graduate students and early-career professionals studying or working in Canada or the United States. It was a multifaceted learning opportunity for participants to expand evaluation capacity building (ECB) competencies and served as a bridge between formal education and real-world practice. This practice note offers reflections from five students representing all teams and one coach on how EC3 supported competency development as outlined by the Canadian Evaluation Society and the American Evaluation Association. Focused on domains related to professional reflection, technical and methodological skills, situational context, as well as management and interpersonal skills, this article explores the role of EC3 in honing skills specific to ECB, preparing evaluators to excel in their roles and champion ECB in diverse and evolving contexts.
{"title":"Developing Evaluation Capacity Building Competencies: Participant Reflections From the Evaluation Capacity Case Challenge","authors":"Amanda Sutter, Michelle Rondeau, Karolina Kaminska, Sandrine Desforges, Sebastian Betzer, Melissa Tremblay","doi":"10.3138/cjpe-2024-0004","DOIUrl":"https://doi.org/10.3138/cjpe-2024-0004","url":null,"abstract":"In 2023, McGill University’s Max Bell School of Public Policy hosted the inaugural Evaluation Capacity Case Challenge (EC3) competition with a cohort of 19 selected graduate students and early-career professionals studying or working in Canada or the United States. It was a multifaceted learning opportunity for participants to expand evaluation capacity building (ECB) competencies and served as a bridge between formal education and real-world practice. This practice note offers reflections from five students representing all teams and one coach on how EC3 supported competency development as outlined by the Canadian Evaluation Society and the American Evaluation Association. Focused on domains related to professional reflection, technical and methodological skills, situational context, as well as management and interpersonal skills, this article explores the role of EC3 in honing skills specific to ECB, preparing evaluators to excel in their roles and champion ECB in diverse and evolving contexts.","PeriodicalId":504262,"journal":{"name":"Canadian Journal of Program Evaluation","volume":"64 25","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140795102","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Past research on evaluation capacity in community organizations has focused on their capacity to conduct rather than use evaluation. This study sought to address this gap by exploring evaluation use and the supporting processes, systems, and structures in the community sector. The authors administered an online survey to 102 individuals working or involved in community organizations to measure their perceptions of their organization’s capacity to use evaluation. The survey was based on an adaptation of Bourgeois and Cousins’ evaluation capacity framework and a related assessment tool, the Organizational Evaluation Capacity Assessment Instrument, for the community sector context. Their findings offer empirical insights into three critical dimensions of the adapted framework associated with evaluation use in community organizations: evaluation literacy, organizational decision-making, and learning benefits. This exploratory study generated several key findings, including the need for tools and policies that establish a strong link between evaluation and decision-making, expanding the scope of engagement in data interpretation beyond primary users, and extending the dissemination of findings to external stakeholders. Implications for practice and research are discussed.
{"title":"Evaluation Literacy and Use in Community Organizations: Insights and Implications for Evaluation Capacity Building","authors":"D. Buetti, Isabelle Bourgeois, Sébastien Savard","doi":"10.3138/cjpe-2024-0002","DOIUrl":"https://doi.org/10.3138/cjpe-2024-0002","url":null,"abstract":"Past research on evaluation capacity in community organizations has focused on their capacity to conduct rather than use evaluation. This study sought to address this gap by exploring evaluation use and the supporting processes, systems, and structures in the community sector. The authors administered an online survey to 102 individuals working or involved in community organizations to measure their perceptions of their organization’s capacity to use evaluation. The survey was based on an adaptation of Bourgeois and Cousins’ evaluation capacity framework and a related assessment tool, the Organizational Evaluation Capacity Assessment Instrument, for the community sector context. Their findings offer empirical insights into three critical dimensions of the adapted framework associated with evaluation use in community organizations: evaluation literacy, organizational decision-making, and learning benefits. This exploratory study generated several key findings, including the need for tools and policies that establish a strong link between evaluation and decision-making, expanding the scope of engagement in data interpretation beyond primary users, and extending the dissemination of findings to external stakeholders. Implications for practice and research are discussed.","PeriodicalId":504262,"journal":{"name":"Canadian Journal of Program Evaluation","volume":"672 ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140787489","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
S. B. Nielsen, Sebastian Lemire, Isabelle Bourgeois, Leslie A. Fierro
This article surveys the literature on organizational evaluation capacity (EC) instruments. Over the past 25 years, articles have been published in four waves covering the development of organizational evaluation capacity models, surveys of the evaluation capacity and practice landscape, the development of EC instruments, and their replication and adaptation. Altogether, the authors identified 20 studies applying 16 EC instruments. Several studies applied variants of the same instrument. We found instruments of three types: checklists, rubrics, and questionnaires. The instruments vary in terms of type, purpose, scoring, dimensions, number of items, and validation processes. In general, the validation studies demonstrated acceptable face, content, construct validity, and internal consistency. Only two studies tested for predictive validity. No studies covered concurrent validity and test–retest, or inter-rater reliability.
{"title":"Capturing Evaluation Capacity: Findings from a Mapping of Evaluation Capacity Instruments","authors":"S. B. Nielsen, Sebastian Lemire, Isabelle Bourgeois, Leslie A. Fierro","doi":"10.3138/cjpe-2024-0001","DOIUrl":"https://doi.org/10.3138/cjpe-2024-0001","url":null,"abstract":"This article surveys the literature on organizational evaluation capacity (EC) instruments. Over the past 25 years, articles have been published in four waves covering the development of organizational evaluation capacity models, surveys of the evaluation capacity and practice landscape, the development of EC instruments, and their replication and adaptation. Altogether, the authors identified 20 studies applying 16 EC instruments. Several studies applied variants of the same instrument. We found instruments of three types: checklists, rubrics, and questionnaires. The instruments vary in terms of type, purpose, scoring, dimensions, number of items, and validation processes. In general, the validation studies demonstrated acceptable face, content, construct validity, and internal consistency. Only two studies tested for predictive validity. No studies covered concurrent validity and test–retest, or inter-rater reliability.","PeriodicalId":504262,"journal":{"name":"Canadian Journal of Program Evaluation","volume":"19 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140775070","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Michelle Searle, Leslie A. Fierro, Jen Pinarski, Laurie Dixon, Melissa Tremblay, Isabelle Bourgeois, R. Gokiert
This practice note shares key learnings from the inaugural Evaluation Capacity Case Challenge (EC3) held by the Max Bell School of Public Policy in April 2023. The purpose is to give readers an opportunity to consider how to shape evaluation capacity in general and within a municipal context. Evaluation capacity building (ECB) is a multifaceted concept to support shared learning about and understanding of evaluation. Strands of learning about ECB emerged from EC3 that can be applied within organizations. These centre on three important actions: (1) establishing a community of practice led by champions, (2) scaling and sequencing evaluation capacity, and (3) developing data literacy as a technical skill. The ECB strategies developed in EC3 offered new ways for the City of Kingston to consider and reconsider how it conducts, shares, learns from, and uses evaluation. Here, the authors describe evaluation within the City of Kingston, the city’s goals in entering EC3, key learnings from the challenge, and the city’s ensuing efforts to propel the growth of evaluation.
{"title":"Meeting the Challenge: How the City of Kingston Is Working to Propel Evaluation Growth","authors":"Michelle Searle, Leslie A. Fierro, Jen Pinarski, Laurie Dixon, Melissa Tremblay, Isabelle Bourgeois, R. Gokiert","doi":"10.3138/cjpe-2024-0006","DOIUrl":"https://doi.org/10.3138/cjpe-2024-0006","url":null,"abstract":"This practice note shares key learnings from the inaugural Evaluation Capacity Case Challenge (EC3) held by the Max Bell School of Public Policy in April 2023. The purpose is to give readers an opportunity to consider how to shape evaluation capacity in general and within a municipal context. Evaluation capacity building (ECB) is a multifaceted concept to support shared learning about and understanding of evaluation. Strands of learning about ECB emerged from EC3 that can be applied within organizations. These centre on three important actions: (1) establishing a community of practice led by champions, (2) scaling and sequencing evaluation capacity, and (3) developing data literacy as a technical skill. The ECB strategies developed in EC3 offered new ways for the City of Kingston to consider and reconsider how it conducts, shares, learns from, and uses evaluation. Here, the authors describe evaluation within the City of Kingston, the city’s goals in entering EC3, key learnings from the challenge, and the city’s ensuing efforts to propel the growth of evaluation.","PeriodicalId":504262,"journal":{"name":"Canadian Journal of Program Evaluation","volume":"31 ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140769622","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
R. Gokiert, Michelle Searle, Kirsty Choquette, Rachel Zukiwsky, Isabelle Bourgeois, Leslie A. Fierro, Melissa Tremblay
There is a growing need for evaluation capacity building (ECB) in community organizations; however, learning opportunities for both organizations and post-secondary students are limited, particularly opportunities to apply evaluation concepts in practice. This practice note describes four initiatives and highlights the ways in which experiential learning can be leveraged through community-university collaborations to provide responsive, hands-on ECB for both students and community organizations. The authors discuss the unique contributions of these initiatives and considerations for designing and implementing similar initiatives.
{"title":"Evaluation Capacity Building: Experiential Learning Through Community–University Collaboratives","authors":"R. Gokiert, Michelle Searle, Kirsty Choquette, Rachel Zukiwsky, Isabelle Bourgeois, Leslie A. Fierro, Melissa Tremblay","doi":"10.3138/cjpe-2024-0003","DOIUrl":"https://doi.org/10.3138/cjpe-2024-0003","url":null,"abstract":"There is a growing need for evaluation capacity building (ECB) in community organizations; however, learning opportunities for both organizations and post-secondary students are limited, particularly opportunities to apply evaluation concepts in practice. This practice note describes four initiatives and highlights the ways in which experiential learning can be leveraged through community-university collaborations to provide responsive, hands-on ECB for both students and community organizations. The authors discuss the unique contributions of these initiatives and considerations for designing and implementing similar initiatives.","PeriodicalId":504262,"journal":{"name":"Canadian Journal of Program Evaluation","volume":"356 4","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140781134","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Editor’s Remarks","authors":"I. Bourgeois","doi":"10.3138/cjpe.35.1.v","DOIUrl":"https://doi.org/10.3138/cjpe.35.1.v","url":null,"abstract":"<jats:p>N/A</jats:p>","PeriodicalId":504262,"journal":{"name":"Canadian Journal of Program Evaluation","volume":"6 7","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141202567","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：