N. Robinson-García, R. Costas, Tina Nane, T. V. van Leeuwen
Evaluation systems have been long criticised for abusing and misusing bibliometric indicators. This has created a culture by which academics are constantly exposing their daily work to the standards they are expected to perform. In this study we investigate whether researchers’ own values and expectations are in line with the expectations of the evaluation system. We conduct a multiple case-study of five departments in two Dutch universities to examine how they balance between their own valuation regimes and the evaluation schemes. For this we combine curriculum analysis with a series of semi-structured interviews. We propose a model to study diversity of academic activities and apply it to the multiple-case study to understand how such diversity is shaped by discipline and career stage. We conclude that the observed misalignment is not only resulting from an abuse of metrics, but also by a lack of tools to evaluate performance in a contextualised and adaptable way.
{"title":"Valuation regimes in academia: Researchers’ attitudes towards their diversity of activities and academic performance","authors":"N. Robinson-García, R. Costas, Tina Nane, T. V. van Leeuwen","doi":"10.31235/osf.io/ve7d3","DOIUrl":"https://doi.org/10.31235/osf.io/ve7d3","url":null,"abstract":"Evaluation systems have been long criticised for abusing and misusing bibliometric indicators. This has created a culture by which academics are constantly exposing their daily work to the standards they are expected to perform. In this study we investigate whether researchers’ own values and expectations are in line with the expectations of the evaluation system. We conduct a multiple case-study of five departments in two Dutch universities to examine how they balance between their own valuation regimes and the evaluation schemes. For this we combine curriculum analysis with a series of semi-structured interviews. We propose a model to study diversity of academic activities and apply it to the multiple-case study to understand how such diversity is shaped by discipline and career stage. We conclude that the observed misalignment is not only resulting from an abuse of metrics, but also by a lack of tools to evaluate performance in a contextualised and adaptable way.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46176573","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The technological innovation of wind power is crucial to energy security and energy structure transformation. The Chinese government has been committed to improving the innovation of the wind power industry for decades. Although academic researchers and wind power policymakers have been widely concerned about the absolute score of innovation output, concentrating on innovation output in isolation is ultimately insufficient. This article goes beyond bean counting and evaluates the innovation of the Chinese wind power industry from the perspective of relative efficiency, and then assesses the efforts of the government to improve innovation efficiency (IE). The study uses the data of 105 wind power listed enterprises in China over the period 2008–2019. According to the recommendation made by the existing papers, IE is defined as the capability to generate innovation outputs per unit of R&D investment. Regression analysis is applied to test the policy effect. During 2008–2019, the average value of IE of Chinese wind power industry is 0.196. The IE of wind power enterprises in eastern China (0.265) is higher than that in central and western China (0.169). Besides, the regression results indicate that all categories of wind power innovation policies contribute to the IE of wind power industry in China. Furthermore, the innovation policies issued by the departments of the State Council significantly improve the IE of wind power industry, but the innovation policies from the National People's Congress and the State Council have no significant impacts on IE. [ FROM AUTHOR] Copyright of Research Evaluation is the property of Oxford University Press / USA and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full . (Copyright applies to all s.)
{"title":"Beyond bean counting: Is the policy effective for the innovation efficiency of wind power industry in China?","authors":"Zihao Jiang, Jiarong Shi, Zhiying Liu, Lei Gong","doi":"10.1093/RESEVAL/RVAB037","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAB037","url":null,"abstract":"The technological innovation of wind power is crucial to energy security and energy structure transformation. The Chinese government has been committed to improving the innovation of the wind power industry for decades. Although academic researchers and wind power policymakers have been widely concerned about the absolute score of innovation output, concentrating on innovation output in isolation is ultimately insufficient. This article goes beyond bean counting and evaluates the innovation of the Chinese wind power industry from the perspective of relative efficiency, and then assesses the efforts of the government to improve innovation efficiency (IE). The study uses the data of 105 wind power listed enterprises in China over the period 2008–2019. According to the recommendation made by the existing papers, IE is defined as the capability to generate innovation outputs per unit of R&D investment. Regression analysis is applied to test the policy effect. During 2008–2019, the average value of IE of Chinese wind power industry is 0.196. The IE of wind power enterprises in eastern China (0.265) is higher than that in central and western China (0.169). Besides, the regression results indicate that all categories of wind power innovation policies contribute to the IE of wind power industry in China. Furthermore, the innovation policies issued by the departments of the State Council significantly improve the IE of wind power industry, but the innovation policies from the National People's Congress and the State Council have no significant impacts on IE. [ FROM AUTHOR] Copyright of Research Evaluation is the property of Oxford University Press / USA and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full . (Copyright applies to all s.)","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-11-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42264110","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Research evaluations for an energy transition? Insights from a review of Swedish research evaluation reports","authors":"S. Sandin, Mats Benner","doi":"10.1093/RESEVAL/RVAB031","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAB031","url":null,"abstract":"","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47927639","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A makerspace is a place where people create artifacts while sharing ideas, equipment, and knowledge. In so doing, makers develop a range of knowledge and skills, such as creativity, problem-solving, collaboration, and self-regulation, to help them achieve their goals. These skills are broadly touted as key for learning and transferable across disciplines and making contexts. This article will first review the state of play in the literature to assess skill development. Secondly, itreports on the trial of an assessment framework developed through a literature review and implemented in a maker learning environment with an elementary school context. Finally, the article concludes with implications for practice.
{"title":"Teachers conceptualizing and developing assessment for skill development: Trialing a maker assessment framework","authors":"J. Lock, Sandra Becker, P. Redmond","doi":"10.1093/reseval/rvab029","DOIUrl":"https://doi.org/10.1093/reseval/rvab029","url":null,"abstract":"\u0000 A makerspace is a place where people create artifacts while sharing ideas, equipment, and knowledge. In so doing, makers develop a range of knowledge and skills, such as creativity, problem-solving, collaboration, and self-regulation, to help them achieve their goals. These skills are broadly touted as key for learning and transferable across disciplines and making contexts. This article will first review the state of play in the literature to assess skill development. Secondly, itreports on the trial of an assessment framework developed through a literature review and implemented in a maker learning environment with an elementary school context. Finally, the article concludes with implications for practice.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43487731","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Intensified pressure to publish is a hallmark of a rapidly evolving higher education field where the faculty of any hue cannot avoid the ‘publish or perish’ treadmill. Growing need to publish more and to do so fast have resulted in the proliferation of pseudo scholarly publications many regards as ‘predatory’. This article provides a systematic review of research studies on so-called ‘predatory’ publishing, a new but fast-growing area of research, with a particular focus on the awareness of prospective authors about so-called ‘predatory’ publishing, the profile of authors publishing in ‘predatory’ journals and the causal factors encouraging authors to publish in such outlets. It synthetizes the results of research studies on the topic to identify gaps and trends in the existing knowledgebase to guide further research. Results indicate so-called ‘predatory’ articles are authored by scholars from all fields and levels of academic experience rather than by inexperienced scholars only and ‘predatory’ contributions are not limited to developing countries, suggesting geographical location and author experience fail to explain the author profile of ‘predatory’ articles. Findings of this review suggest causal factors include research evaluation policies and publication pressure that emerge from the research environment in which scholars operate authors’ limited capacity to publish in ‘legitimate’ journals and conventions of so-called ‘predatory’ publishers. This indicates meaningful action might address all these factors in combination, rather than focus on them in isolation.
{"title":"Profile of authors publishing in ‘predatory’ journals and causal factors behind their decision: A systematic review","authors":"Sefika Mertkan, Gülen Onurkan Aliusta, Nilgün Suphi","doi":"10.1093/reseval/rvab032","DOIUrl":"https://doi.org/10.1093/reseval/rvab032","url":null,"abstract":"\u0000 Intensified pressure to publish is a hallmark of a rapidly evolving higher education field where the faculty of any hue cannot avoid the ‘publish or perish’ treadmill. Growing need to publish more and to do so fast have resulted in the proliferation of pseudo scholarly publications many regards as ‘predatory’. This article provides a systematic review of research studies on so-called ‘predatory’ publishing, a new but fast-growing area of research, with a particular focus on the awareness of prospective authors about so-called ‘predatory’ publishing, the profile of authors publishing in ‘predatory’ journals and the causal factors encouraging authors to publish in such outlets. It synthetizes the results of research studies on the topic to identify gaps and trends in the existing knowledgebase to guide further research. Results indicate so-called ‘predatory’ articles are authored by scholars from all fields and levels of academic experience rather than by inexperienced scholars only and ‘predatory’ contributions are not limited to developing countries, suggesting geographical location and author experience fail to explain the author profile of ‘predatory’ articles. Findings of this review suggest causal factors include research evaluation policies and publication pressure that emerge from the research environment in which scholars operate authors’ limited capacity to publish in ‘legitimate’ journals and conventions of so-called ‘predatory’ publishers. This indicates meaningful action might address all these factors in combination, rather than focus on them in isolation.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-09-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45139918","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract This paper assesses the risk of two COVID-19 related changes necessary for the expert-review of the REF2021’s Impact criterion: the move from F2F to virtual deliberation; and the changing research landscape caused by the COVID-19 crisis requiring an extension of deadlines, and accommodation of COVID-19 related mitigation. Peer review in its basic form requires expert debate, where dissenting opinions and non-verbal cues are absorbed into a groups deliberative practice and therefore inform outcomes. With a move to deliberations in virtual settings, the most likely current outcome for REF2021 evaluations, the extent that negotiation dynamics necessary in F2F evaluations are diminished and how this limits panelists’ ability to sensitively assess COVID-19 mitigation statements is questioned. This article explores the nature of, and associated capabilities to undertake, complex decision making in virtual settings around the Impact criterion as well the consequences of COVID-19 on normal Impact trajectories. It examines the risks these changes present for evaluation of the Impact criterion and provides recommendations to offset these risks to enhance discussion and safeguard the legitimacy of evaluation outcomes. This paper is also relevant for evaluation processes of academic criteria that require both a shift to virtual, and/or guidance of how to sensitively assess the effect of COVID-19 on narratives of individual, group or organisational performance.
{"title":"The Corona-Eye: Exploring the risks of COVID-19 on fair assessments of impact for REF 2021","authors":"G. Derrick, J. Bayley","doi":"10.1093/reseval/rvab033","DOIUrl":"https://doi.org/10.1093/reseval/rvab033","url":null,"abstract":"Abstract This paper assesses the risk of two COVID-19 related changes necessary for the expert-review of the REF2021’s Impact criterion: the move from F2F to virtual deliberation; and the changing research landscape caused by the COVID-19 crisis requiring an extension of deadlines, and accommodation of COVID-19 related mitigation. Peer review in its basic form requires expert debate, where dissenting opinions and non-verbal cues are absorbed into a groups deliberative practice and therefore inform outcomes. With a move to deliberations in virtual settings, the most likely current outcome for REF2021 evaluations, the extent that negotiation dynamics necessary in F2F evaluations are diminished and how this limits panelists’ ability to sensitively assess COVID-19 mitigation statements is questioned. This article explores the nature of, and associated capabilities to undertake, complex decision making in virtual settings around the Impact criterion as well the consequences of COVID-19 on normal Impact trajectories. It examines the risks these changes present for evaluation of the Impact criterion and provides recommendations to offset these risks to enhance discussion and safeguard the legitimacy of evaluation outcomes. This paper is also relevant for evaluation processes of academic criteria that require both a shift to virtual, and/or guidance of how to sensitively assess the effect of COVID-19 on narratives of individual, group or organisational performance.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"1 1","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41492423","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：