Abstract There is increasing attention on evidencing research impact and applying a systems thinking perspective in public health. However, there is limited understanding of the extent to which and how public health research that applies a systems thinking perspective contributes to changes in system behaviour and improved population health outcomes. This paper addresses the theoretical limitations of research impact, theory-based evaluation and systems thinking, by drawing on their respective literature to develop an initial, middle-range Theory of Systems Change, focused on the contribution of public health research that takes a systems perspective on population health outcomes. The Theory of Systems Change was developed through four phases: (1) Preliminary activities, (2) Theory development, (3) Scripting into images, and (4) Examining against Merton’s criteria. The primary propositions are: that well-functioning systems create the conditions for improved population health outcomes; the inter-related properties of, and practices within, well-functioning systems include adaptation, alignment, collaboration and evidence-driven action and learning; and public health research contributes to population health outcomes by embedding capacity in the system. The Theory of Systems Change can guide researchers in developing project-specific theories of change and creates the theoretical architecture for the accumulation of learning. The Theory of Systems Change is necessarily incomplete and an initial attempt to develop a theory to be scrutinized and tested. Ultimately, it seeks to advance theory and provide evidence-based guidance to maximize the contribution of research. We provide examples of how we have applied the Theory of Systems Change to Pathways in Place.
{"title":"Theory of systems change: An initial, middle-range theory of public health research impact","authors":"Melinda Craike, Bojana Klepac, Amy Mowle, Therese Riley","doi":"10.1093/reseval/rvad030","DOIUrl":"https://doi.org/10.1093/reseval/rvad030","url":null,"abstract":"Abstract There is increasing attention on evidencing research impact and applying a systems thinking perspective in public health. However, there is limited understanding of the extent to which and how public health research that applies a systems thinking perspective contributes to changes in system behaviour and improved population health outcomes. This paper addresses the theoretical limitations of research impact, theory-based evaluation and systems thinking, by drawing on their respective literature to develop an initial, middle-range Theory of Systems Change, focused on the contribution of public health research that takes a systems perspective on population health outcomes. The Theory of Systems Change was developed through four phases: (1) Preliminary activities, (2) Theory development, (3) Scripting into images, and (4) Examining against Merton’s criteria. The primary propositions are: that well-functioning systems create the conditions for improved population health outcomes; the inter-related properties of, and practices within, well-functioning systems include adaptation, alignment, collaboration and evidence-driven action and learning; and public health research contributes to population health outcomes by embedding capacity in the system. The Theory of Systems Change can guide researchers in developing project-specific theories of change and creates the theoretical architecture for the accumulation of learning. The Theory of Systems Change is necessarily incomplete and an initial attempt to develop a theory to be scrutinized and tested. Ultimately, it seeks to advance theory and provide evidence-based guidance to maximize the contribution of research. We provide examples of how we have applied the Theory of Systems Change to Pathways in Place.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135728959","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract This study examines the impact of the Fraunhofer Society, Europe’s largest network of applied research institutes, on patent applications. A difference-in-differences strategy was employed exploiting the establishment of five new Fraunhofer centres in the 2000s. The panel includes 65,963 European applicants (both firms and independent inventors) between 1980 and 2019. The results show that establishing a centre increases patent output by at least 13%, robust to using applicants of cities that established a centre by the end of the 2010s as an alternative control group. The effect is driven by an increase in applicants’ productivity and not by agglomeration dynamics.
{"title":"The effect of applied research institutes on invention: Evidence from the Fraunhofer centres in Europe","authors":"Pedro Llanos-Paredes","doi":"10.1093/reseval/rvad028","DOIUrl":"https://doi.org/10.1093/reseval/rvad028","url":null,"abstract":"Abstract This study examines the impact of the Fraunhofer Society, Europe’s largest network of applied research institutes, on patent applications. A difference-in-differences strategy was employed exploiting the establishment of five new Fraunhofer centres in the 2000s. The panel includes 65,963 European applicants (both firms and independent inventors) between 1980 and 2019. The results show that establishing a centre increases patent output by at least 13%, robust to using applicants of cities that established a centre by the end of the 2010s as an alternative control group. The effect is driven by an increase in applicants’ productivity and not by agglomeration dynamics.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135854396","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract Impact assessment research has developed theory-based approaches to trace the societal impact of scientific research. Impact assessment typically starts from the perspective of a research investment, organization, or project. Research users, non-academic actors involved in knowledge production, translation, and application, are well represented in many of these approaches. Researcher users are usually positioned as contributors to research, recipients of research outputs, or beneficiaries of research-driven outcomes. This paper argues that impact assessment would benefit from a more comprehensive understanding and analysis of research valorization processes from the user perspective. The first half of the paper reviews key impact assessment literature to identify how research users are positioned and portrayed in relation to valorization processes. In the second half of the paper, we use the results of this review to propose a set of principles to guide a systematic approach to constructing user perspectives on research impact. We suggest four concepts for operationalization of this approach. The paper concludes that the addition of a more comprehensive research user perspective on research valorization would complement and enhance existing impact assessment approaches.
{"title":"Research impact seen from the user side","authors":"Richard Woolley, Jordi Molas-Gallart","doi":"10.1093/reseval/rvad027","DOIUrl":"https://doi.org/10.1093/reseval/rvad027","url":null,"abstract":"Abstract Impact assessment research has developed theory-based approaches to trace the societal impact of scientific research. Impact assessment typically starts from the perspective of a research investment, organization, or project. Research users, non-academic actors involved in knowledge production, translation, and application, are well represented in many of these approaches. Researcher users are usually positioned as contributors to research, recipients of research outputs, or beneficiaries of research-driven outcomes. This paper argues that impact assessment would benefit from a more comprehensive understanding and analysis of research valorization processes from the user perspective. The first half of the paper reviews key impact assessment literature to identify how research users are positioned and portrayed in relation to valorization processes. In the second half of the paper, we use the results of this review to propose a set of principles to guide a systematic approach to constructing user perspectives on research impact. We suggest four concepts for operationalization of this approach. The paper concludes that the addition of a more comprehensive research user perspective on research valorization would complement and enhance existing impact assessment approaches.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135476872","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract This paper contributes to understanding the effects of research governance on global scientific fields. Using a highly selective comparative analysis of four national governance contexts, we explore how governance arrangements influence the dynamics of global research fields. Our study provides insights into second-level governance effects, moving beyond previous studies focusing primarily on effects on research organizations rooted in national contexts. Rather than study over 100 countries across which our selected CERN-based particle physics global research field operates, we explore conditions for changing the dynamics of global research fields and examine mechanisms through which change may occur. We predict then minimal effects on the epistemic choices and research practices of members of the four local knowledge networks despite variations in governance arrangements, and hence no second-level effects. We assert a research field’s independence from governance depends on its characteristics and the relative importance to researchers of research quality notions. This paper contributes methodologically and has practical implications for policymakers. It suggests governance arrangements affect the epistemic choices and research practices of the local knowledge networks only when certain conditions are met. Policymakers should consider the context and characteristics of a field when designing governance arrangements and policy.
{"title":"Aggregate level research governance effects on particle physics: A comparative analysis","authors":"Mayra M Tirado, Maria Nedeva, Duncan A Thomas","doi":"10.1093/reseval/rvad025","DOIUrl":"https://doi.org/10.1093/reseval/rvad025","url":null,"abstract":"Abstract This paper contributes to understanding the effects of research governance on global scientific fields. Using a highly selective comparative analysis of four national governance contexts, we explore how governance arrangements influence the dynamics of global research fields. Our study provides insights into second-level governance effects, moving beyond previous studies focusing primarily on effects on research organizations rooted in national contexts. Rather than study over 100 countries across which our selected CERN-based particle physics global research field operates, we explore conditions for changing the dynamics of global research fields and examine mechanisms through which change may occur. We predict then minimal effects on the epistemic choices and research practices of members of the four local knowledge networks despite variations in governance arrangements, and hence no second-level effects. We assert a research field’s independence from governance depends on its characteristics and the relative importance to researchers of research quality notions. This paper contributes methodologically and has practical implications for policymakers. It suggests governance arrangements affect the epistemic choices and research practices of the local knowledge networks only when certain conditions are met. Policymakers should consider the context and characteristics of a field when designing governance arrangements and policy.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135476862","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract The paper takes a scientometric approach to measure the change in research productivity of Italian academics before the COVID-19 pandemic outbreak. We propose a composite output/input bibliometric indicator and apply it at the field level, conducting a longitudinal analysis. Although the number of academics in the national academic system has decreased, we register very strong growth in both the number of publications and their scholarly impact. The growth in productivity, with only rare exceptions, crosses almost all fields. However, in areas that are traditionally very internationalized (Biology, Physics, and Chemistry), growth is less sustained than overall average, and also the variability of productivity across fields seems reduced. The main reason for this detail would be the smaller margins for improvement in the fields that had already reached high international standing. What emerges from the analysis goes counter to some alarms of declining scientific productivity at the global level. The Italian case is partly explained by the historic adoption of policies aimed at strengthening competitive mechanisms, in particular through the introduction of systems of performance-based research funding, and bibliometric accreditation for professorship.
{"title":"Warnings of declining research productivity: Does Italy buck the trend?","authors":"Giovanni Abramo, Ciriaco Andrea D’Angelo","doi":"10.1093/reseval/rvad026","DOIUrl":"https://doi.org/10.1093/reseval/rvad026","url":null,"abstract":"Abstract The paper takes a scientometric approach to measure the change in research productivity of Italian academics before the COVID-19 pandemic outbreak. We propose a composite output/input bibliometric indicator and apply it at the field level, conducting a longitudinal analysis. Although the number of academics in the national academic system has decreased, we register very strong growth in both the number of publications and their scholarly impact. The growth in productivity, with only rare exceptions, crosses almost all fields. However, in areas that are traditionally very internationalized (Biology, Physics, and Chemistry), growth is less sustained than overall average, and also the variability of productivity across fields seems reduced. The main reason for this detail would be the smaller margins for improvement in the fields that had already reached high international standing. What emerges from the analysis goes counter to some alarms of declining scientific productivity at the global level. The Italian case is partly explained by the historic adoption of policies aimed at strengthening competitive mechanisms, in particular through the introduction of systems of performance-based research funding, and bibliometric accreditation for professorship.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135258461","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Thomas Zacharewicz, Noemi Pulido Pavón, Luis Antonio Palma Martos, Benedetto Lepori
Abstract Over the last decades, most EU countries have profoundly reshaped their public research funding systems by shifting from traditional institutional block-funding towards more project-based mechanisms. The main rationale underlying this evolution builds on the assumption that project funding would foster research performance through the introduction of competitive allocation mechanisms. In contrast with the general increase of project funding, evidence is mixed regarding a positive effect of competitive funding mechanisms on research performance, as some studies find a positive impact, other a negative one or no impact. Differences also appear across studies regarding research actors, funding streams, and research outputs considered. This article integrates these different approaches through a multilevel design gathering funding inputs for 10 countries and 148 universities between 2011 and 2019 and assesses their impact on the quantity and quality of publications. Results highlight no impact of national and university-level competitive funding mechanisms on universities highly cited publications and no clear effect on the quantity of publications.
{"title":"Do funding modes matter? A multilevel analysis of funding allocation mechanisms on university research performance","authors":"Thomas Zacharewicz, Noemi Pulido Pavón, Luis Antonio Palma Martos, Benedetto Lepori","doi":"10.1093/reseval/rvad023","DOIUrl":"https://doi.org/10.1093/reseval/rvad023","url":null,"abstract":"Abstract Over the last decades, most EU countries have profoundly reshaped their public research funding systems by shifting from traditional institutional block-funding towards more project-based mechanisms. The main rationale underlying this evolution builds on the assumption that project funding would foster research performance through the introduction of competitive allocation mechanisms. In contrast with the general increase of project funding, evidence is mixed regarding a positive effect of competitive funding mechanisms on research performance, as some studies find a positive impact, other a negative one or no impact. Differences also appear across studies regarding research actors, funding streams, and research outputs considered. This article integrates these different approaches through a multilevel design gathering funding inputs for 10 countries and 148 universities between 2011 and 2019 and assesses their impact on the quantity and quality of publications. Results highlight no impact of national and university-level competitive funding mechanisms on universities highly cited publications and no clear effect on the quantity of publications.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"335 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135451533","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Although ex post evaluation of impact is increasingly common, the extent to which research impacts emerge largely as anticipated by researchers, or as the result of serendipitous and unpredictable processes, is not well understood. In this article, we explore whether predictions of impact made at the funding stage align with realized impact, using data from the UK’s Research Excellence Framework (REF). We exploit REF impact cases traced back to research funding applications, as a dataset of 2,194 case–grant pairs, to compare impact topics with funder remits. For 209 of those pairs, we directly compare their descriptions of ex ante and ex post impact. We find that impact claims in these case–grant pairs are often congruent with each other, with 76% showing alignment between anticipated impact at funding stage and the eventual claimed impact in the REF. Co-production of research, often perceived as a model for impactful research, was a feature of just over half of our cases. Our results show that, contrary to other preliminary studies of the REF, impact appears to be broadly predictable, although unpredictability remains important. We suggest that co-production is a reasonably good mechanism for addressing the balance of predictable and unpredictable impact outcomes.
{"title":"How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework","authors":"O. Yaqub, D. Malkov, Josh Siepel","doi":"10.1093/reseval/rvad019","DOIUrl":"https://doi.org/10.1093/reseval/rvad019","url":null,"abstract":"\u0000 Although ex post evaluation of impact is increasingly common, the extent to which research impacts emerge largely as anticipated by researchers, or as the result of serendipitous and unpredictable processes, is not well understood. In this article, we explore whether predictions of impact made at the funding stage align with realized impact, using data from the UK’s Research Excellence Framework (REF). We exploit REF impact cases traced back to research funding applications, as a dataset of 2,194 case–grant pairs, to compare impact topics with funder remits. For 209 of those pairs, we directly compare their descriptions of ex ante and ex post impact. We find that impact claims in these case–grant pairs are often congruent with each other, with 76% showing alignment between anticipated impact at funding stage and the eventual claimed impact in the REF. Co-production of research, often perceived as a model for impactful research, was a feature of just over half of our cases. Our results show that, contrary to other preliminary studies of the REF, impact appears to be broadly predictable, although unpredictability remains important. We suggest that co-production is a reasonably good mechanism for addressing the balance of predictable and unpredictable impact outcomes.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-07-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49479173","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Federico Vasen, N. Sarthou, Silvina A. Romano, Brenda D Gutiérrez, Manuel Pintos
Evaluation procedures play a crucial role in science and technology systems, particularly within academic career structures. This article focuses on an approach to evaluation that has gained prominence in Latin America over the past four decades. This scheme assesses the individual performance of academics based on their academic activities and outputs and assigns them a ‘researcher category’, which carries prestige and, in many cases, additional monthly rewards. Initially implemented in higher education contexts with limited research focus, these systems aimed to bolster knowledge production by involving more academics in research. In this study, we define National Researcher Categorization Systems (NRCSs) and distinguish them from other evaluation systems. Subsequently, we present a comparative analysis of NRCSs in seven countries, identifying common trends. Additionally, we discuss categorization systems within the broader context of strategies employed to incentivize academic research, and we explore the potential structural effects that arise when NRCSs assume a central role in a science system. Through our research, we have identified a family of systems in five countries (Mexico, Argentina, Uruguay, Paraguay, and Panama) that share a common history and structure. Furthermore, we emphasize that NRCSs may reinforce a traditional model of the academic researcher, potentially impeding the development of professional profiles aligned with research directed toward social objectives. In summary, our study sheds light on NRCSs, providing insights into their nature, comparative analysis across countries, and implications within the broader academic research landscape.
{"title":"Turning academics into researchers: The development of National Researcher Categorization Systems in Latin America","authors":"Federico Vasen, N. Sarthou, Silvina A. Romano, Brenda D Gutiérrez, Manuel Pintos","doi":"10.1093/reseval/rvad021","DOIUrl":"https://doi.org/10.1093/reseval/rvad021","url":null,"abstract":"\u0000 Evaluation procedures play a crucial role in science and technology systems, particularly within academic career structures. This article focuses on an approach to evaluation that has gained prominence in Latin America over the past four decades. This scheme assesses the individual performance of academics based on their academic activities and outputs and assigns them a ‘researcher category’, which carries prestige and, in many cases, additional monthly rewards. Initially implemented in higher education contexts with limited research focus, these systems aimed to bolster knowledge production by involving more academics in research. In this study, we define National Researcher Categorization Systems (NRCSs) and distinguish them from other evaluation systems. Subsequently, we present a comparative analysis of NRCSs in seven countries, identifying common trends. Additionally, we discuss categorization systems within the broader context of strategies employed to incentivize academic research, and we explore the potential structural effects that arise when NRCSs assume a central role in a science system. Through our research, we have identified a family of systems in five countries (Mexico, Argentina, Uruguay, Paraguay, and Panama) that share a common history and structure. Furthermore, we emphasize that NRCSs may reinforce a traditional model of the academic researcher, potentially impeding the development of professional profiles aligned with research directed toward social objectives. In summary, our study sheds light on NRCSs, providing insights into their nature, comparative analysis across countries, and implications within the broader academic research landscape.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-07-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44318941","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Peer review is a key gatekeeper for academic journals, attempting to block inadequate submissions or correcting them to a publishable standard, as well as improving those that are already satisfactory. The three key aspects of research quality are rigour, significance, and originality but no prior study has assessed whether journal reviewers are ever able to judge these effectively. In response, this article compares reviewer scores for these aspects for theoretical articles in the SciPost Physics journal. It also compares them with Italian research assessment exercise physics reviewer agreement scores. SciPost Physics theoretical articles give a nearly ideal case: a theoretical aspect of a mature science, for which suitable reviewers might comprehend the entire paper. Nevertheless, intraclass correlations between the first two reviewers for the three core quality scores were similar and moderate, 0.36 (originality), 0.39 (significance), and 0.40 (rigour), so there is no aspect that different reviewers are consistent about. Differences tended to be small, with 86% of scores agreeing or differing by 1 on a 6-point scale. Individual reviewers were most likely to give similar scores for significance and originality (Spearman 0.63), and least likely to for originality and validity (Spearman 0.38). Whilst a lack of norm referencing is probably the biggest reason for differences between reviewers, others include differing background knowledge, understanding, and beliefs about valid assumptions. The moderate agreement between reviewers on the core aspects of scientific quality, including rigour, in a nearly ideal case is concerning for the security of the wider academic record.
{"title":"Can journal reviewers dependably assess rigour, significance, and originality in theoretical papers? Evidence from physics","authors":"M. Thelwall, J. Hołyst","doi":"10.1093/reseval/rvad018","DOIUrl":"https://doi.org/10.1093/reseval/rvad018","url":null,"abstract":"\u0000 Peer review is a key gatekeeper for academic journals, attempting to block inadequate submissions or correcting them to a publishable standard, as well as improving those that are already satisfactory. The three key aspects of research quality are rigour, significance, and originality but no prior study has assessed whether journal reviewers are ever able to judge these effectively. In response, this article compares reviewer scores for these aspects for theoretical articles in the SciPost Physics journal. It also compares them with Italian research assessment exercise physics reviewer agreement scores. SciPost Physics theoretical articles give a nearly ideal case: a theoretical aspect of a mature science, for which suitable reviewers might comprehend the entire paper. Nevertheless, intraclass correlations between the first two reviewers for the three core quality scores were similar and moderate, 0.36 (originality), 0.39 (significance), and 0.40 (rigour), so there is no aspect that different reviewers are consistent about. Differences tended to be small, with 86% of scores agreeing or differing by 1 on a 6-point scale. Individual reviewers were most likely to give similar scores for significance and originality (Spearman 0.63), and least likely to for originality and validity (Spearman 0.38). Whilst a lack of norm referencing is probably the biggest reason for differences between reviewers, others include differing background knowledge, understanding, and beliefs about valid assumptions. The moderate agreement between reviewers on the core aspects of scientific quality, including rigour, in a nearly ideal case is concerning for the security of the wider academic record.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-06-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45098607","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
E. Giménez-Toledo, Julia Olmos-Peñuela, Elena Castro-Martínez, François Perruchas
Science policymakers are devoting increasing attention to enhancing the social valorization of scientific knowledge. Since 2010, several international evaluation initiatives have been implemented to assess knowledge transfer and exchange practices and the societal impacts of research. Analysis of these initiatives would allow investigation of the different knowledge transfer and exchange channels and their effects on society and how their effects could be evaluated and boosted. The present study analyses the transfer sexenio programme, which is a first (pilot) assessment that was conducted in Spain to evaluate the engagement of individual researchers in knowledge transfer to and knowledge exchange with non-academic stakeholders, including professionals and society at large. The breadth of the information and supporting documentation available (more than 16,000 applications and 81,000 contributions) allows an exploration of knowledge valorization practices in terms of the transfer forms used and the researchers involved—distinguishing between the Science, Technology, Engineering and Mathematics (STEM) and Social Sciences, Humanities and Arts (SSHA) areas. By focusing on SSHA fields, we explore knowledge dissemination via enlightenment or professional outputs. We conduct quantitative and qualitative analysis which provide a more comprehensive overview of knowledge transfer practices in Spain in the SSHA field, in particular, and has implications for future assessment exercises.
{"title":"The forms of societal interaction in the social sciences, humanities and arts: Below the tip of the iceberg","authors":"E. Giménez-Toledo, Julia Olmos-Peñuela, Elena Castro-Martínez, François Perruchas","doi":"10.1093/reseval/rvad016","DOIUrl":"https://doi.org/10.1093/reseval/rvad016","url":null,"abstract":"\u0000 Science policymakers are devoting increasing attention to enhancing the social valorization of scientific knowledge. Since 2010, several international evaluation initiatives have been implemented to assess knowledge transfer and exchange practices and the societal impacts of research. Analysis of these initiatives would allow investigation of the different knowledge transfer and exchange channels and their effects on society and how their effects could be evaluated and boosted. The present study analyses the transfer sexenio programme, which is a first (pilot) assessment that was conducted in Spain to evaluate the engagement of individual researchers in knowledge transfer to and knowledge exchange with non-academic stakeholders, including professionals and society at large. The breadth of the information and supporting documentation available (more than 16,000 applications and 81,000 contributions) allows an exploration of knowledge valorization practices in terms of the transfer forms used and the researchers involved—distinguishing between the Science, Technology, Engineering and Mathematics (STEM) and Social Sciences, Humanities and Arts (SSHA) areas. By focusing on SSHA fields, we explore knowledge dissemination via enlightenment or professional outputs. We conduct quantitative and qualitative analysis which provide a more comprehensive overview of knowledge transfer practices in Spain in the SSHA field, in particular, and has implications for future assessment exercises.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48229891","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：