{"title":"Correction to: Methods for measuring social and conceptual dimensions of convergence science","authors":"","doi":"10.1093/reseval/rvad036","DOIUrl":"https://doi.org/10.1093/reseval/rvad036","url":null,"abstract":"","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"45 22","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-12-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138592151","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Correction to: Stated preference methods and STI policy studies: a foreground approach","authors":"","doi":"10.1093/reseval/rvad037","DOIUrl":"https://doi.org/10.1093/reseval/rvad037","url":null,"abstract":"","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"42 19","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-12-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138593870","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A tribute to our dearly departed colleague and friend: An introduction to the Special Issue in memory of Prof. Paul Benneworth","authors":"G. E. Derrick, J. Olmos-Peñuela, S. D. de Jong","doi":"10.1093/reseval/rvad010","DOIUrl":"https://doi.org/10.1093/reseval/rvad010","url":null,"abstract":"","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"24 1","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-11-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139214837","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
During the past decade, responsible research assessment (RRA) has become a major science policy goal to advance responsible research and innovation and open science. Starting with the DORA declaration in 2012, common understanding of the needs and demands of RRA has been shaped by a growing number of initiatives, culminating in a European Commission supported Agreement on reforming research assessment published in July 2022. In this paper, we outline and investigate the normative framework for RRA, ranging from binding legal norms (hard law), rules of conduct which, in principle, have no legally binding force but which nevertheless may have practical effects’ (soft law) to self-regulation in supranational (the European Union) and one national (Italy) context. As shown by our analysis of hard and soft legal sources for RRA, apart from constitutional and legal provisions regarding freedom of research, autonomy of universities and non-discrimination and gender equality, the legal foundation for RRA rests mainly on soft law and policy norms. In the discussion we relate the normative framework for RRA to research integrity guidelines and consider the challenges of ensuring coherent RRA implementation across countries and research funding and performing institutions. We conclude that self-regulation and soft law are good starting points and may pave the way for hard law down the line: this can be seen as the optimum mix of regulatory mechanisms to mainstream RRAs.
{"title":"The legal foundation of responsible research assessment: An overview on European Union and Italy","authors":"G. Peruginelli, Janne Pölönen","doi":"10.1093/reseval/rvad035","DOIUrl":"https://doi.org/10.1093/reseval/rvad035","url":null,"abstract":"During the past decade, responsible research assessment (RRA) has become a major science policy goal to advance responsible research and innovation and open science. Starting with the DORA declaration in 2012, common understanding of the needs and demands of RRA has been shaped by a growing number of initiatives, culminating in a European Commission supported Agreement on reforming research assessment published in July 2022. In this paper, we outline and investigate the normative framework for RRA, ranging from binding legal norms (hard law), rules of conduct which, in principle, have no legally binding force but which nevertheless may have practical effects’ (soft law) to self-regulation in supranational (the European Union) and one national (Italy) context. As shown by our analysis of hard and soft legal sources for RRA, apart from constitutional and legal provisions regarding freedom of research, autonomy of universities and non-discrimination and gender equality, the legal foundation for RRA rests mainly on soft law and policy norms. In the discussion we relate the normative framework for RRA to research integrity guidelines and consider the challenges of ensuring coherent RRA implementation across countries and research funding and performing institutions. We conclude that self-regulation and soft law are good starting points and may pave the way for hard law down the line: this can be seen as the optimum mix of regulatory mechanisms to mainstream RRAs.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"204 1","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-11-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139220180","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Marta Natalia Wróblewska, Corina Balaban, Gemma Derrick, Paul Benneworth
It has been argued that due to the growing importance attributed to research impact and forms of its evaluation, an academic ‘culture of impact’ is emerging. It would include certain concepts, values, and skills related to the area of generating and documenting impact. We use thematic and discourse analysis to analyse open answers from 100 questionnaires on research impact submitted by ECRs working in the social sciences and humanities (SSH) in Europe. We explore ECR’s early-career stage positions relative to societal impact and the trade-offs necessary to assure an academic career. The results show how, as the first generation of scholars to be socialized towards value of academic research beyond academia, ECRs are confronted with policy signals that encourage a drive for impact, which are at the same time often in line with respondents’ personal values around impact beyond academia. However, ECRs face a number of competing signals about research value within the evaluation spaces necessary to navigate an academic career. Current evaluative structures often dismiss the achievement of societal impact favouring instead narrower definitions of research excellence. Career structures and organizational realities are often unfavourable to impact-related activity, which has implications for an ECRs’ ability to develop coherent professional positionings.
{"title":"The conflict of impact for early career researchers planning for a future in the academy","authors":"Marta Natalia Wróblewska, Corina Balaban, Gemma Derrick, Paul Benneworth","doi":"10.1093/reseval/rvad024","DOIUrl":"https://doi.org/10.1093/reseval/rvad024","url":null,"abstract":"It has been argued that due to the growing importance attributed to research impact and forms of its evaluation, an academic ‘culture of impact’ is emerging. It would include certain concepts, values, and skills related to the area of generating and documenting impact. We use thematic and discourse analysis to analyse open answers from 100 questionnaires on research impact submitted by ECRs working in the social sciences and humanities (SSH) in Europe. We explore ECR’s early-career stage positions relative to societal impact and the trade-offs necessary to assure an academic career. The results show how, as the first generation of scholars to be socialized towards value of academic research beyond academia, ECRs are confronted with policy signals that encourage a drive for impact, which are at the same time often in line with respondents’ personal values around impact beyond academia. However, ECRs face a number of competing signals about research value within the evaluation spaces necessary to navigate an academic career. Current evaluative structures often dismiss the achievement of societal impact favouring instead narrower definitions of research excellence. Career structures and organizational realities are often unfavourable to impact-related activity, which has implications for an ECRs’ ability to develop coherent professional positionings.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"35 26","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-11-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139252593","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract This paper addresses policy effects of international mobility in developing countries. It proposes a multilevel approach to study research training policies, specifically focusing on international mobility schemes as capacity-building instruments, where effects need to be identified at both individual and organizational levels. The paper asserts that current categorizations of country efforts to build a solid scientific base should consider transference of effects in domestic research contexts. Using a survey and interview-based study of a mobility policy in Mexico, and nanotechnology-related disciplines and sectors, this paper’s findings suggest that mobility policies are crucial in creating high-quality individual research skills, but that their impact on building domestic research capacity may be modest. This paper contributes to the evaluation literature on national research training programs and their influence on research capacity in two ways. First, it mobilizes a multilevel analytical approach to examine the effects of policies on individuals and organizations, offering a broader conceptualization of impact. Second, by introducing these individual and organizational effects, the paper provides insights into how effects are enacted and transferred.
{"title":"Individual and organizational effects of international mobility policy: Effects of Mexico’s Scholarship Program in nanotechnology","authors":"Mayra M Tirado","doi":"10.1093/reseval/rvad031","DOIUrl":"https://doi.org/10.1093/reseval/rvad031","url":null,"abstract":"Abstract This paper addresses policy effects of international mobility in developing countries. It proposes a multilevel approach to study research training policies, specifically focusing on international mobility schemes as capacity-building instruments, where effects need to be identified at both individual and organizational levels. The paper asserts that current categorizations of country efforts to build a solid scientific base should consider transference of effects in domestic research contexts. Using a survey and interview-based study of a mobility policy in Mexico, and nanotechnology-related disciplines and sectors, this paper’s findings suggest that mobility policies are crucial in creating high-quality individual research skills, but that their impact on building domestic research capacity may be modest. This paper contributes to the evaluation literature on national research training programs and their influence on research capacity in two ways. First, it mobilizes a multilevel analytical approach to examine the effects of policies on individuals and organizations, offering a broader conceptualization of impact. Second, by introducing these individual and organizational effects, the paper provides insights into how effects are enacted and transferred.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" 520","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135186623","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Gemma Elizabeth Derrick, Alessandra Zimmermann, Helen Greaves, Jonathan Best, Richard Klavans
Abstract Previous studies of the use of peer review for the allocation of competitive funding agencies have concentrated on questions of efficiency and how to make the ‘best’ decision, by ensuring that successful applicants are also the more productive or visible in the long term. This paper examines the components of feedback received from an unsuccessful grant application, is associated with motivating applicants career decisions to persist (reapply for funding at T1), or to switch (not to reapply, or else leave academia). This study combined data from interviews with unsuccessful ECR applicants (n = 19) to The Wellcome Trust 2009–19, and manual coding of reviewer comments received by applicants (n = 81). All applicants received feedback on their application at T0 with a large proportion of unsuccessful applicants reapplying for funding at T1. Here, peer-review-comments-as-feedback sends signals to applicants to encourage them to persist (continue) or switch (not continue) even when the initial application has failed. Feedback associated by unsuccessful applicants as motivating their decision to resubmit had three characteristics: actionable; targeted; and fair. The results lead to identification of standards of feedback for funding agencies and peer-reviewers to promote when providing reviewer feedback to applicants as part of their peer review process. The provision of quality reviewer-reports-as-feedback to applicants, ensures that peer review acts as a participatory research governance tool focused on supporting the development of individuals and their future research plans.
{"title":"Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices","authors":"Gemma Elizabeth Derrick, Alessandra Zimmermann, Helen Greaves, Jonathan Best, Richard Klavans","doi":"10.1093/reseval/rvad034","DOIUrl":"https://doi.org/10.1093/reseval/rvad034","url":null,"abstract":"Abstract Previous studies of the use of peer review for the allocation of competitive funding agencies have concentrated on questions of efficiency and how to make the ‘best’ decision, by ensuring that successful applicants are also the more productive or visible in the long term. This paper examines the components of feedback received from an unsuccessful grant application, is associated with motivating applicants career decisions to persist (reapply for funding at T1), or to switch (not to reapply, or else leave academia). This study combined data from interviews with unsuccessful ECR applicants (n = 19) to The Wellcome Trust 2009–19, and manual coding of reviewer comments received by applicants (n = 81). All applicants received feedback on their application at T0 with a large proportion of unsuccessful applicants reapplying for funding at T1. Here, peer-review-comments-as-feedback sends signals to applicants to encourage them to persist (continue) or switch (not continue) even when the initial application has failed. Feedback associated by unsuccessful applicants as motivating their decision to resubmit had three characteristics: actionable; targeted; and fair. The results lead to identification of standards of feedback for funding agencies and peer-reviewers to promote when providing reviewer feedback to applicants as part of their peer review process. The provision of quality reviewer-reports-as-feedback to applicants, ensures that peer review acts as a participatory research governance tool focused on supporting the development of individuals and their future research plans.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"13 3","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135976131","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract Transdisciplinary research (TDR) is conceptualized as not only providing societal effects but also benefiting academia. However, recent literature on the evaluation of TDR has focused almost entirely on the societal effects of TDR. A discussion of the scientific effects of TDR is needed to do justice to the potential of this research mode. To date, little empirical research has focused on the effects of TDR on science. Our explorative study addresses this gap. The empirical basis are qualitative interviews with scientists engaged in transdisciplinary research and anchored in three sub-disciplines: environmental sociology, sustainable chemistry, and participatory health research. We identify as main effects of the transdisciplinary research mode: changes in the understanding of scientific problems, changes in the quality of scientific insights, and the promotion of a reflexive turn in science.
{"title":"Effects of transdisciplinary research on scientific knowledge and reflexivity","authors":"Oskar Marg, Lena Theiler","doi":"10.1093/reseval/rvad033","DOIUrl":"https://doi.org/10.1093/reseval/rvad033","url":null,"abstract":"Abstract Transdisciplinary research (TDR) is conceptualized as not only providing societal effects but also benefiting academia. However, recent literature on the evaluation of TDR has focused almost entirely on the societal effects of TDR. A discussion of the scientific effects of TDR is needed to do justice to the potential of this research mode. To date, little empirical research has focused on the effects of TDR on science. Our explorative study addresses this gap. The empirical basis are qualitative interviews with scientists engaged in transdisciplinary research and anchored in three sub-disciplines: environmental sociology, sustainable chemistry, and participatory health research. We identify as main effects of the transdisciplinary research mode: changes in the understanding of scientific problems, changes in the quality of scientific insights, and the promotion of a reflexive turn in science.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"124 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135510075","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Alejandra Boni, Diana Velasco, Jordi Molas-Gallart, Johan Schot
Abstract This article presents the insights from an evaluation of a transformative policy experiment, in the food domain, conducted at the Swedish Agency of Innovation (Vinnova). To be consistent with the principles and objectives of these policies, it was necessary to implement a formative evaluation approach developed in the Transformative Innovation Policy Consortium (TIPC), that is supportive of experimental policies designed for highly complex and uncertain environments. The article presents the main characteristics of this approach and its differences and similarities with other proposals to assess transformative innovation policies. Next, the article describes how the evaluation was conducted in Vinnova during a 2-year engagement. Subsequently, the main insights gleaned from the process are highlighted in terms of (1) understanding the expectations of actors; (2) what has been learned by doing, from a technical and organizational point of view; and (3) the constraints that emerged during the engagement. It is then argued that the TIPC- formative evaluation approach challenges the role of evaluators, stretching them, and requires different attitudes from those that are common in evaluation practice.
{"title":"Evaluating transformative innovation policy in a formative way: Insights from Vinnova’s food mission experiment","authors":"Alejandra Boni, Diana Velasco, Jordi Molas-Gallart, Johan Schot","doi":"10.1093/reseval/rvad029","DOIUrl":"https://doi.org/10.1093/reseval/rvad029","url":null,"abstract":"Abstract This article presents the insights from an evaluation of a transformative policy experiment, in the food domain, conducted at the Swedish Agency of Innovation (Vinnova). To be consistent with the principles and objectives of these policies, it was necessary to implement a formative evaluation approach developed in the Transformative Innovation Policy Consortium (TIPC), that is supportive of experimental policies designed for highly complex and uncertain environments. The article presents the main characteristics of this approach and its differences and similarities with other proposals to assess transformative innovation policies. Next, the article describes how the evaluation was conducted in Vinnova during a 2-year engagement. Subsequently, the main insights gleaned from the process are highlighted in terms of (1) understanding the expectations of actors; (2) what has been learned by doing, from a technical and organizational point of view; and (3) the constraints that emerged during the engagement. It is then argued that the TIPC- formative evaluation approach challenges the role of evaluators, stretching them, and requires different attitudes from those that are common in evaluation practice.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"1 3","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136160057","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract Peer review has developed over time to become the established procedure for assessing and assuring the scientific quality of research. Nevertheless, the procedure has also been variously criticized as conservative, biased, and unfair, among other things. Do scientists regard all these flaws as equally problematic? Do they have the same opinions on which problems are so serious that other selection procedures ought to be considered? The answers to these questions hints at what should be modified in peer review processes as a priority objective. The authors of this paper use survey data to examine how members of the scientific community weight different shortcomings of peer review processes. Which of those processes’ problems do they consider less relevant? Which problems, on the other hand, do they judge to be beyond remedy? Our investigation shows that certain defects of peer review processes are indeed deemed irreparable: (1) legitimate quandaries in the process of fine-tuning the choice between equally eligible research proposals and in the selection of daring ideas; and (2) illegitimate problems due to networks. Science-policy measures to improve peer review processes should therefore clarify the distinction between field-specific remediable and irremediable flaws than is currently the case.
{"title":"Peer review’s irremediable flaws: Scientists’ perspectives on grant evaluation in Germany","authors":"Eva Barlösius, Laura Paruschke, Axel Philipps","doi":"10.1093/reseval/rvad032","DOIUrl":"https://doi.org/10.1093/reseval/rvad032","url":null,"abstract":"Abstract Peer review has developed over time to become the established procedure for assessing and assuring the scientific quality of research. Nevertheless, the procedure has also been variously criticized as conservative, biased, and unfair, among other things. Do scientists regard all these flaws as equally problematic? Do they have the same opinions on which problems are so serious that other selection procedures ought to be considered? The answers to these questions hints at what should be modified in peer review processes as a priority objective. The authors of this paper use survey data to examine how members of the scientific community weight different shortcomings of peer review processes. Which of those processes’ problems do they consider less relevant? Which problems, on the other hand, do they judge to be beyond remedy? Our investigation shows that certain defects of peer review processes are indeed deemed irreparable: (1) legitimate quandaries in the process of fine-tuning the choice between equally eligible research proposals and in the selection of daring ideas; and (2) illegitimate problems due to networks. Science-policy measures to improve peer review processes should therefore clarify the distinction between field-specific remediable and irremediable flaws than is currently the case.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"22 6","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135413139","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：