首页 > 最新文献

Research Evaluation最新文献

英文 中文
Describing the state of a research network: A mixed methods approach to network evaluation. 描述研究网络的状态:网络评估的混合方法。
IF 3.3 4区 管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Pub Date : 2022-10-28 eCollection Date: 2023-04-01 DOI: 10.1093/reseval/rvac034
James M Bowen, Mathieu Ouimet, Justin Lawarée, Joanna Bielecki, Ashley Rhéaume, Caylee Greenberg, Valeria E Rac

Diabetes Action Canada Strategy for Patient-Oriented Research (SPOR) Network in Chronic Disease was formed in 2016 and is funded primarily through the Canadian Institutes of Health Research (CIHR). We propose a novel mixed-methods approach to a network evaluation integrating the State of Network Evaluation framework and the Canadian Academy of Health Sciences (CAHS) preferred framework and indicators. We measure key network themes of connectivity, health and results, and impact and return on investment associated with health research networks. Our methods consist of a longitudinal cross-sectional network survey of members and social network analysis to examine Network Connectivity and assess the frequency of interactions, the topics discussed during them, and how networking effectively facilitates interactions and collaboration among members. Network Health will be evaluated through semistructured interviews, a membership survey inquiring about satisfaction and experience with the Network, and a review of documentary sources related to funding and infrastructure to evaluate Network Sustainability. Finally, we will examine Network Results and Impact using the CAHS preferred framework and indicators to measure returns on investment in health research across the five domains of the CAHS framework, which include: advancing knowledge, capacity building, informing decision making, health impact, and economic and social impact. Indicators will be assessed with various methods, including bibliometric analyses, review of relevant documentary sources (annual reports), member activities informing health and research policy, and Patient Partner involvement. The Network Evaluation will provide members and stakeholders with information for planning, improvements, and funding future Network endeavors.

糖尿病行动加拿大慢性病患者导向研究战略(SPOR)网络成立于2016年,主要由加拿大卫生研究院(CIHR)资助。我们提出了一种新的混合方法来进行网络评估,该方法结合了网络状态评估框架和加拿大卫生科学院(CAHS)的首选框架和指标。我们衡量连接、健康和结果等关键网络主题,以及与健康研究网络相关的投资影响和回报。我们的方法包括对成员的纵向横截面网络调查和社交网络分析,以检查网络连接并评估互动频率、互动期间讨论的主题,以及网络如何有效地促进成员之间的互动和协作。网络健康将通过半结构化访谈、成员调查(询问对网络的满意度和经验)以及审查与资金和基础设施相关的文件来源来评估网络可持续性。最后,我们将使用CAHS首选框架和指标来衡量CAHS框架五个领域的健康研究投资回报,包括:推进知识、能力建设、为决策提供信息、健康影响以及经济和社会影响,来审查网络结果和影响。指标将采用各种方法进行评估,包括文献计量分析、相关文献来源审查(年度报告)、为健康和研究政策提供信息的成员活动以及患者合作伙伴的参与。网络评估将为成员和利益相关者提供规划、改进和资助未来网络工作的信息。
{"title":"Describing the state of a research network: A mixed methods approach to network evaluation.","authors":"James M Bowen,&nbsp;Mathieu Ouimet,&nbsp;Justin Lawarée,&nbsp;Joanna Bielecki,&nbsp;Ashley Rhéaume,&nbsp;Caylee Greenberg,&nbsp;Valeria E Rac","doi":"10.1093/reseval/rvac034","DOIUrl":"10.1093/reseval/rvac034","url":null,"abstract":"<p><p>Diabetes Action Canada Strategy for Patient-Oriented Research (SPOR) Network in Chronic Disease was formed in 2016 and is funded primarily through the Canadian Institutes of Health Research (CIHR). We propose a novel mixed-methods approach to a network evaluation integrating the State of Network Evaluation framework and the Canadian Academy of Health Sciences (CAHS) preferred framework and indicators. We measure key network themes of connectivity, health and results, and impact and return on investment associated with health research networks. Our methods consist of a longitudinal cross-sectional network survey of members and social network analysis to examine Network Connectivity and assess the frequency of interactions, the topics discussed during them, and how networking effectively facilitates interactions and collaboration among members. Network Health will be evaluated through semistructured interviews, a membership survey inquiring about satisfaction and experience with the Network, and a review of documentary sources related to funding and infrastructure to evaluate Network Sustainability. Finally, we will examine Network Results and Impact using the CAHS preferred framework and indicators to measure returns on investment in health research across the five domains of the CAHS framework, which include: advancing knowledge, capacity building, informing decision making, health impact, and economic and social impact. Indicators will be assessed with various methods, including bibliometric analyses, review of relevant documentary sources (annual reports), member activities informing health and research policy, and Patient Partner involvement. The Network Evaluation will provide members and stakeholders with information for planning, improvements, and funding future Network endeavors.</p>","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"32 2","pages":"188-199"},"PeriodicalIF":3.3,"publicationDate":"2022-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10550251/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41173475","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Correction to: Interdisciplinary research and policy impacts: Assessing the significance of knowledge coproduction 修正:跨学科研究和政策影响:评估知识合作生产的重要性
IF 3.3 4区 管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Pub Date : 2022-10-19 DOI: 10.1093/reseval/rvac039
{"title":"Correction to: Interdisciplinary research and policy impacts: Assessing the significance of knowledge coproduction","authors":"","doi":"10.1093/reseval/rvac039","DOIUrl":"https://doi.org/10.1093/reseval/rvac039","url":null,"abstract":"","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2022-10-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44926924","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Explaining employment sector choices of doctoral graduates in Germany 解读德国博士毕业生的就业选择
IF 3.3 4区 管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Pub Date : 2022-10-03 DOI: 10.1093/reseval/rvac030
Lea Goldan, Steffen Jaksztat, C. Gross
Previous research in different national contexts has shown that individual preferences for certain job attributes, objective labour market conditions, subjective career prospects, and external encouragement shape doctoral graduates’ career decisions. For Germany, where the number of awarded doctoral degrees is highest within the European Union and where no established academic tenure-track system exists, the determinants of doctoral graduates’ sector choices are still largely unexplored. This article aims to shed light on the determinants of sector choices of doctoral graduates in Germany. By deriving the determinants from the wide version of rational choice theory and by measuring the determinants prior to employment sectors, we overcome the conceptual and methodological limitations of previous research. Using data from a nationally representative panel survey with doctoral graduates of the 2014 cohort in Germany, we differentiate between five distinct employment sectors and carry out multinomial logistic regression analysis. As expected, and in line with previous research from other countries, the results confirm that the sector choices of doctoral graduates in Germany depend on their preferences as well as various objective and subjective constraints. The article helps to better understand how doctoral graduates select into different employment sectors and thus provides important insights into postdoctoral career trajectories.
先前在不同国家背景下的研究表明,个人对某些工作属性的偏好、客观的劳动力市场条件、主观的职业前景和外部鼓励影响了博士毕业生的职业决策。在德国,授予博士学位的人数是欧盟最高的,而且没有既定的学术终身制,博士毕业生选择行业的决定因素在很大程度上仍未得到探索。本文旨在阐明德国博士毕业生行业选择的决定因素。通过从广泛版本的理性选择理论中推导决定因素,并在就业部门之前测量决定因素,我们克服了以往研究的概念和方法局限性。利用对德国2014届博士毕业生进行的具有全国代表性的小组调查数据,我们区分了五个不同的就业部门,并进行了多项逻辑回归分析。正如预期的那样,与其他国家先前的研究一致,研究结果证实,德国博士毕业生的行业选择取决于他们的偏好以及各种客观和主观约束。这篇文章有助于更好地了解博士毕业生如何选择不同的就业部门,从而为博士后职业轨迹提供重要见解。
{"title":"Explaining employment sector choices of doctoral graduates in Germany","authors":"Lea Goldan, Steffen Jaksztat, C. Gross","doi":"10.1093/reseval/rvac030","DOIUrl":"https://doi.org/10.1093/reseval/rvac030","url":null,"abstract":"\u0000 Previous research in different national contexts has shown that individual preferences for certain job attributes, objective labour market conditions, subjective career prospects, and external encouragement shape doctoral graduates’ career decisions. For Germany, where the number of awarded doctoral degrees is highest within the European Union and where no established academic tenure-track system exists, the determinants of doctoral graduates’ sector choices are still largely unexplored. This article aims to shed light on the determinants of sector choices of doctoral graduates in Germany. By deriving the determinants from the wide version of rational choice theory and by measuring the determinants prior to employment sectors, we overcome the conceptual and methodological limitations of previous research. Using data from a nationally representative panel survey with doctoral graduates of the 2014 cohort in Germany, we differentiate between five distinct employment sectors and carry out multinomial logistic regression analysis. As expected, and in line with previous research from other countries, the results confirm that the sector choices of doctoral graduates in Germany depend on their preferences as well as various objective and subjective constraints. The article helps to better understand how doctoral graduates select into different employment sectors and thus provides important insights into postdoctoral career trajectories.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2022-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46824119","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Research governance and the dynamics of science: A framework for the study of governance effects on research fields 研究治理与科学动态:研究治理对研究领域影响的框架
IF 3.3 4区 管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Pub Date : 2022-08-19 DOI: 10.1093/reseval/rvac028
M. Nedeva, Mayra M. Tirado, Duncan A. Thomas
This article offers a framework for the study of research governance effects on scientific fields framed by notions of research quality and the epistemic, organizational, and career choices they entail. The framework interprets the contested idea of ‘quality’ as an interplay involving notion origins, quality attributes, and contextual sites. We mobilize the origin and site components, to frame organizational-level events where quality notions inform selections, or selection events. Through the dynamic interplay between notions selected at specific sites, we contend, local actors enact research quality cumulatively, by making choices that privilege certain notions over others. In this article, we contribute in four ways. First, we propose an approach to study research governance effects on scientific fields. Second, we introduce first- and second-level effects of research governance paving the way to identify mechanisms through which these different levels of effects occur. Third, we assert that interactions between research spaces and fields leading to effects occur in the context of research organizations, and at nine key selection events. Fourth, and lastly, we discuss an empirical test on an illustration case to demonstrate how this approach can be applied.
本文提供了一个框架来研究研究治理对科学领域的影响,其框架是研究质量的概念及其所带来的认知、组织和职业选择。该框架将有争议的“质量”概念解释为涉及概念起源、质量属性和上下文站点的相互作用。我们调动起源和地点组成部分,以制定组织级别的事件,其中质量概念为选择或选择事件提供信息。我们认为,通过在特定地点选择的概念之间的动态相互作用,当地行为者通过做出使某些概念优先于其他概念的选择,积累了研究质量。在本文中,我们从四个方面进行了贡献。首先,我们提出了一种研究治理对科学领域影响的方法。其次,我们介绍了研究治理的一级和二级效应,为确定产生这些不同水平效应的机制铺平了道路。第三,我们断言,导致效应的研究空间和领域之间的互动发生在研究组织的背景下,以及在九个关键的选择事件中。第四,也是最后一点,我们讨论了一个例证案例的实证检验,以证明如何应用这种方法。
{"title":"Research governance and the dynamics of science: A framework for the study of governance effects on research fields","authors":"M. Nedeva, Mayra M. Tirado, Duncan A. Thomas","doi":"10.1093/reseval/rvac028","DOIUrl":"https://doi.org/10.1093/reseval/rvac028","url":null,"abstract":"\u0000 This article offers a framework for the study of research governance effects on scientific fields framed by notions of research quality and the epistemic, organizational, and career choices they entail. The framework interprets the contested idea of ‘quality’ as an interplay involving notion origins, quality attributes, and contextual sites. We mobilize the origin and site components, to frame organizational-level events where quality notions inform selections, or selection events. Through the dynamic interplay between notions selected at specific sites, we contend, local actors enact research quality cumulatively, by making choices that privilege certain notions over others. In this article, we contribute in four ways. First, we propose an approach to study research governance effects on scientific fields. Second, we introduce first- and second-level effects of research governance paving the way to identify mechanisms through which these different levels of effects occur. Third, we assert that interactions between research spaces and fields leading to effects occur in the context of research organizations, and at nine key selection events. Fourth, and lastly, we discuss an empirical test on an illustration case to demonstrate how this approach can be applied.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2022-08-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45780294","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
The impact of Italian performance-based research funding systems on the intensity of international research collaboration 意大利基于绩效的研究资助制度对国际研究合作强度的影响
IF 3.3 4区 管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Pub Date : 2022-07-29 DOI: 10.1093/reseval/rvac026
G. Abramo, Ciriaco Andrea D’Angelo
The study of national research assessment exercises serves to evaluate the effectiveness of policies versus their objectives and to improve the formulation of future initiatives. The aim of the current study is to verify whether the introduction of the first performance-based research funding in Italy, based on the 2004–10 VQR assessment, achieved the intended objective of inducing greater international collaboration on the part of researchers. For this, we apply a bibliometric approach based on the observation of coauthorships in Italian and worldwide scientific publications over a 14-year period, beginning in the target years of the VQR assessment. Through an Interrupted Time Series Analysis, we compare the expected and observed patterns of international coauthorship for Italy and the rest of the world. Although the rate of internationalization of Italian research is seen to be increasing, whether this is a consequence of the VQR incentives, or rather part of a global phenomenon of recourse to international collaboration in response to the increasingly complex scientific challenges, it is open to interpretation.
对国家研究评估工作的研究有助于评估政策与其目标的有效性,并改进未来举措的制定。本研究的目的是验证意大利在2004–10年VQR评估的基础上引入的第一笔基于绩效的研究资金是否实现了促使研究人员加强国际合作的预期目标。为此,我们采用了一种文献计量方法,该方法基于对意大利和世界各地科学出版物的合著者在14年内的观察,从VQR评估的目标年份开始。通过中断时间序列分析,我们比较了意大利和世界其他地区预期和观察到的国际合著模式。尽管意大利研究的国际化程度正在提高,但无论这是VQR激励措施的结果,还是为了应对日益复杂的科学挑战而诉诸国际合作的全球现象的一部分,都可以解释。
{"title":"The impact of Italian performance-based research funding systems on the intensity of international research collaboration","authors":"G. Abramo, Ciriaco Andrea D’Angelo","doi":"10.1093/reseval/rvac026","DOIUrl":"https://doi.org/10.1093/reseval/rvac026","url":null,"abstract":"\u0000 The study of national research assessment exercises serves to evaluate the effectiveness of policies versus their objectives and to improve the formulation of future initiatives. The aim of the current study is to verify whether the introduction of the first performance-based research funding in Italy, based on the 2004–10 VQR assessment, achieved the intended objective of inducing greater international collaboration on the part of researchers. For this, we apply a bibliometric approach based on the observation of coauthorships in Italian and worldwide scientific publications over a 14-year period, beginning in the target years of the VQR assessment. Through an Interrupted Time Series Analysis, we compare the expected and observed patterns of international coauthorship for Italy and the rest of the world. Although the rate of internationalization of Italian research is seen to be increasing, whether this is a consequence of the VQR incentives, or rather part of a global phenomenon of recourse to international collaboration in response to the increasingly complex scientific challenges, it is open to interpretation.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2022-07-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47425606","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 2
Tilting at twin windmills: On article quotas and journal impact factors 在双风车上倾斜:论文章配额和期刊影响因素
IF 3.3 4区 管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Pub Date : 2022-07-28 DOI: 10.1093/reseval/rvac024
J. Serafy
{"title":"Tilting at twin windmills: On article quotas and journal impact factors","authors":"J. Serafy","doi":"10.1093/reseval/rvac024","DOIUrl":"https://doi.org/10.1093/reseval/rvac024","url":null,"abstract":"","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2022-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49292670","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Evaluation of the arts in performance-based research funding systems: An international perspective 基于绩效的研究资助体系中的艺术评估:国际视角
IF 3.3 4区 管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Pub Date : 2022-07-18 DOI: 10.1093/reseval/rvac017
Kamila Lewandowska, Emanuel Kulczycki, M. Ochsner
This article provides a comprehensive analysis of the evaluation of the arts within performance-based research funding systems (PRFSs). Previous literature on PRFSs has overlooked the arts and focussed primarily on outputs in relation to the sciences and humanities. We develop a typology of how artistic outputs are evaluated within 10 countries’ PRFSs, operating in Australia, the Czech Republic, Italy, Lithuania, New Zealand, Poland, Portugal, Slovakia, Spain, and the UK, and identify three different types of artistic evaluation systems. The study compares evaluation methods and provides a classification of quality criteria used by evaluation panels. We conclude with a discussion of the challenges specific to different types of systems.
本文对基于绩效的研究资助体系(PRFS)中的艺术评估进行了全面分析。以前关于PRFS的文献忽略了艺术,主要关注与科学和人文学科相关的产出。我们在澳大利亚、捷克共和国、意大利、立陶宛、新西兰、波兰、葡萄牙、斯洛伐克、西班牙和英国的10个国家的PRFS中制定了艺术产出评估类型,并确定了三种不同类型的艺术评估系统。该研究比较了评估方法,并对评估小组使用的质量标准进行了分类。最后,我们讨论了不同类型系统所特有的挑战。
{"title":"Evaluation of the arts in performance-based research funding systems: An international perspective","authors":"Kamila Lewandowska, Emanuel Kulczycki, M. Ochsner","doi":"10.1093/reseval/rvac017","DOIUrl":"https://doi.org/10.1093/reseval/rvac017","url":null,"abstract":"\u0000 This article provides a comprehensive analysis of the evaluation of the arts within performance-based research funding systems (PRFSs). Previous literature on PRFSs has overlooked the arts and focussed primarily on outputs in relation to the sciences and humanities. We develop a typology of how artistic outputs are evaluated within 10 countries’ PRFSs, operating in Australia, the Czech Republic, Italy, Lithuania, New Zealand, Poland, Portugal, Slovakia, Spain, and the UK, and identify three different types of artistic evaluation systems. The study compares evaluation methods and provides a classification of quality criteria used by evaluation panels. We conclude with a discussion of the challenges specific to different types of systems.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2022-07-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45288797","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 2
Quis judicabit ipsos judices? A case study on the dynamics of competitive funding panel evaluations 为什么是jusicabit ipsos jusices ?关于竞争性供资小组评价动态的个案研究
IF 3.3 4区 管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Pub Date : 2022-07-15 DOI: 10.1093/reseval/rvac021
João M. Santos
Securing research funding is essential for all researchers. The standard evaluation method for competitive grants is through evaluation by a panel of experts. However, the literature notes that peer review has inherent flaws and is subject to biases, which can arise from differing interpretations of the criteria, the impossibility for a group of reviewers to be experts in all possible topics within their field, and the role of affect. As such, understanding the dynamics at play during panel evaluations is crucial to allow researchers a better chance at securing funding, and also for the reviewers themselves to be aware of the cognitive mechanisms underlying their decision-making. In this study, we conduct a case study based on application and evaluation data for two social sciences panels in a competitive state-funded call in Portugal. Using a mixed-methods approach, we find that qualitative evaluations largely resonate with the evaluation criteria, and the candidate’s scientific output is partially aligned with the qualitative evaluations, but scientometric indicators alone do not significantly influence the candidate’s evaluation. However, the polarity of the qualitative evaluation has a positive influence on the candidate’s evaluation. This paradox is discussed as possibly resulting from the occurrence of a halo effect in the panel’s judgment of the candidates. By providing a multi-methods approach, this study aims to provide insights that can be useful for all stakeholders involved in competitive funding evaluations.
确保研究经费对所有研究人员都至关重要。竞争性赠款的标准评估方法是由一个专家小组进行评估。然而,文献指出,同行评议有其固有的缺陷,并受到偏见的影响,这可能源于对标准的不同解释,一组评议者不可能在其领域内所有可能的主题上都是专家,以及情感的作用。因此,了解小组评估过程中的动态是至关重要的,这可以让研究人员有更好的机会获得资助,也可以让审稿人自己意识到他们决策背后的认知机制。在本研究中,我们对两个社会科学小组在葡萄牙竞争性国家资助项目中的应用和评估数据进行了案例研究。采用混合方法,我们发现定性评价与评价标准基本一致,候选人的科学产出与定性评价部分一致,但单独的科学计量指标对候选人的评价没有显著影响。然而,定性评价的极性对候选人的评价有积极的影响。这一悖论被认为可能是由于评委会对候选人的判断中出现了光环效应而引起的。通过提供多方法方法,本研究旨在为参与竞争性资金评估的所有利益相关者提供有用的见解。
{"title":"Quis judicabit ipsos judices? A case study on the dynamics of competitive funding panel evaluations","authors":"João M. Santos","doi":"10.1093/reseval/rvac021","DOIUrl":"https://doi.org/10.1093/reseval/rvac021","url":null,"abstract":"\u0000 Securing research funding is essential for all researchers. The standard evaluation method for competitive grants is through evaluation by a panel of experts. However, the literature notes that peer review has inherent flaws and is subject to biases, which can arise from differing interpretations of the criteria, the impossibility for a group of reviewers to be experts in all possible topics within their field, and the role of affect. As such, understanding the dynamics at play during panel evaluations is crucial to allow researchers a better chance at securing funding, and also for the reviewers themselves to be aware of the cognitive mechanisms underlying their decision-making. In this study, we conduct a case study based on application and evaluation data for two social sciences panels in a competitive state-funded call in Portugal. Using a mixed-methods approach, we find that qualitative evaluations largely resonate with the evaluation criteria, and the candidate’s scientific output is partially aligned with the qualitative evaluations, but scientometric indicators alone do not significantly influence the candidate’s evaluation. However, the polarity of the qualitative evaluation has a positive influence on the candidate’s evaluation. This paradox is discussed as possibly resulting from the occurrence of a halo effect in the panel’s judgment of the candidates. By providing a multi-methods approach, this study aims to provide insights that can be useful for all stakeholders involved in competitive funding evaluations.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2022-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44940876","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Valorization of transdisciplinary research: An evaluation approach and empirical illustration 跨学科研究的价值化:一种评估方法与实证说明
IF 3.3 4区 管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Pub Date : 2022-06-30 DOI: 10.1093/reseval/rvac019
Stefania Munaretto, C. Mooren, L. Hessels
In recent times, there has been a surge of impact-oriented, transdisciplinary research programmes and projects integrating multiple disciplines, types of knowledge and practices. An essential element often mentioned in the literature to improve the performance of these programmes and support impact delivery is continuous reflection and learning via evaluation. We argue that because a standard format for organizing transdisciplinary research does not exist, tailor-made evaluation approaches designed around the specificities of each programme are needed. The existing evaluation literature provides useful building blocks that can be integrated and adapted to specific transdisciplinary research contexts. In this article, we develop the valorization cycle and apply it to evaluate a transdisciplinary research programme in the water sector. The building blocks of our approach are: understanding of valorization as a cyclical process; theory of change as a logic model to structure the evaluation; productive interactions as a process indicator articulated in learning outcomes (cognitive, relational, and strategic); and impact pathways as narratives to explain impact dynamics. Our framework is based both on research evaluation and learning literature and on our personal experience in the evaluation of transdisciplinary research. The evaluation of the programme showed how the valorization cycle can point to different learning outcomes across the research process that are conducive to impact, and provided useful insights to the programme managers to adjust the programme. The principle of tailoring an evaluation approach to the specificities of the programme evaluated using building blocks from the literature will be applicable in other transdisciplinary contexts too.
近年来,融合了多个学科、知识类型和实践的注重影响的跨学科研究方案和项目激增。文献中经常提到的一个重要因素是通过评估不断反思和学习,以提高这些方案的绩效并支持影响的交付。我们认为,由于不存在组织跨学科研究的标准格式,因此需要围绕每个方案的具体情况设计量身定制的评估方法。现有的评估文献提供了有用的构建块,可以整合并适应特定的跨学科研究背景。在本文中,我们开发了定价周期,并将其应用于评估水务部门的跨学科研究计划。我们的方法的组成部分是:将定价理解为一个周期性过程;变革理论作为构建评价的逻辑模型;生产性互动作为学习结果(认知、关系和战略)中表达的过程指标;以及作为解释影响动态的叙述的影响途径。我们的框架基于研究评估和学习文献,以及我们在跨学科研究评估中的个人经验。对该方案的评估表明,定价周期如何指向整个研究过程中有利于产生影响的不同学习结果,并为方案管理人员调整该方案提供了有用的见解。使用文献中的构建块根据评估方案的具体情况调整评估方法的原则也适用于其他跨学科背景。
{"title":"Valorization of transdisciplinary research: An evaluation approach and empirical illustration","authors":"Stefania Munaretto, C. Mooren, L. Hessels","doi":"10.1093/reseval/rvac019","DOIUrl":"https://doi.org/10.1093/reseval/rvac019","url":null,"abstract":"\u0000 In recent times, there has been a surge of impact-oriented, transdisciplinary research programmes and projects integrating multiple disciplines, types of knowledge and practices. An essential element often mentioned in the literature to improve the performance of these programmes and support impact delivery is continuous reflection and learning via evaluation. We argue that because a standard format for organizing transdisciplinary research does not exist, tailor-made evaluation approaches designed around the specificities of each programme are needed. The existing evaluation literature provides useful building blocks that can be integrated and adapted to specific transdisciplinary research contexts. In this article, we develop the valorization cycle and apply it to evaluate a transdisciplinary research programme in the water sector. The building blocks of our approach are: understanding of valorization as a cyclical process; theory of change as a logic model to structure the evaluation; productive interactions as a process indicator articulated in learning outcomes (cognitive, relational, and strategic); and impact pathways as narratives to explain impact dynamics. Our framework is based both on research evaluation and learning literature and on our personal experience in the evaluation of transdisciplinary research. The evaluation of the programme showed how the valorization cycle can point to different learning outcomes across the research process that are conducive to impact, and provided useful insights to the programme managers to adjust the programme. The principle of tailoring an evaluation approach to the specificities of the programme evaluated using building blocks from the literature will be applicable in other transdisciplinary contexts too.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2022-06-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44778904","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Under pressure: The extent and distribution of perceived pressure among scientists in Germany, Austria, and Switzerland 压力下:德国、奥地利和瑞士科学家感知压力的程度和分布
IF 3.3 4区 管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE Pub Date : 2022-06-21 DOI: 10.1093/reseval/rvac014
David Johann, Isabel J. Raabe, H. Rauhut
While it has been stressed repeatedly that academics nowadays have come to face extensive pressure, the extent and distribution of pressure to publish and to secure third-party funding has not been systematically investigated on a large scale. Based on the Zurich Survey of Academics, a representative large-scale web survey among academics working at universities in Germany, Austria, and Switzerland (DACH region), which measures perceived pressure using six-point Likert scales, this article examines the extent and distribution of pressure to publish and to attract external funding. Specifically, we examine differences in perceived pressure across countries, disciplines, types of higher education institutions, sociodemographic characteristics (academic status, age, gender, relationship status, number of children), and working conditions (non-tenured/tenured, time available for research). It is shown that researchers in the DACH region feel a high level of pressure, with the pressure to publish being slightly greater than the pressure to attract external funding. The results also suggest that perceived pressure is not evenly distributed among countries and groups of academics. Specifically, the results suggest that (1) more secure and permanent (tenured) positions should be created and (2) the high-performance culture should be addressed in at least some disciplines if excessive pressure is to be alleviated. The findings also suggest that further investments should be made (3) to promote equal career opportunities for women and men and (4) to ensure that scientists have sufficient time for their research and are not too occupied with other responsibilities, such as teaching or administrative duties.
虽然人们一再强调,现在的学者面临着巨大的压力,但发表论文和获得第三方资助的压力的程度和分布却没有得到大规模的系统调查。基于苏黎世学者调查,这是一项在德国、奥地利和瑞士(DACH地区)大学工作的学者中具有代表性的大规模网络调查,使用六点李克特量表测量感知压力,本文研究了出版和吸引外部资金的压力的程度和分布。具体来说,我们研究了不同国家、学科、高等教育机构类型、社会人口特征(学术地位、年龄、性别、关系状况、子女数量)和工作条件(非终身教职/终身教职、可用于研究的时间)在感知压力方面的差异。研究表明,DACH地区的研究人员感受到较高的压力,发表论文的压力略大于吸引外部资助的压力。研究结果还表明,感知到的压力在不同国家和学者群体之间的分布并不均匀。具体而言,研究结果表明:(1)应该创造更多安全的、永久的(终身)职位;(2)如果要缓解过度的压力,至少应该在某些学科中建立高绩效文化。研究结果还表明,应该进一步投资(3)促进男女平等的职业机会,(4)确保科学家有足够的时间进行研究,而不是被其他责任占据,比如教学或行政职责。
{"title":"Under pressure: The extent and distribution of perceived pressure among scientists in Germany, Austria, and Switzerland","authors":"David Johann, Isabel J. Raabe, H. Rauhut","doi":"10.1093/reseval/rvac014","DOIUrl":"https://doi.org/10.1093/reseval/rvac014","url":null,"abstract":"\u0000 While it has been stressed repeatedly that academics nowadays have come to face extensive pressure, the extent and distribution of pressure to publish and to secure third-party funding has not been systematically investigated on a large scale. Based on the Zurich Survey of Academics, a representative large-scale web survey among academics working at universities in Germany, Austria, and Switzerland (DACH region), which measures perceived pressure using six-point Likert scales, this article examines the extent and distribution of pressure to publish and to attract external funding. Specifically, we examine differences in perceived pressure across countries, disciplines, types of higher education institutions, sociodemographic characteristics (academic status, age, gender, relationship status, number of children), and working conditions (non-tenured/tenured, time available for research). It is shown that researchers in the DACH region feel a high level of pressure, with the pressure to publish being slightly greater than the pressure to attract external funding. The results also suggest that perceived pressure is not evenly distributed among countries and groups of academics. Specifically, the results suggest that (1) more secure and permanent (tenured) positions should be created and (2) the high-performance culture should be addressed in at least some disciplines if excessive pressure is to be alleviated. The findings also suggest that further investments should be made (3) to promote equal career opportunities for women and men and (4) to ensure that scientists have sufficient time for their research and are not too occupied with other responsibilities, such as teaching or administrative duties.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2022-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46993665","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
期刊
Research Evaluation
全部 Acc. Chem. Res. ACS Applied Bio Materials ACS Appl. Electron. Mater. ACS Appl. Energy Mater. ACS Appl. Mater. Interfaces ACS Appl. Nano Mater. ACS Appl. Polym. Mater. ACS BIOMATER-SCI ENG ACS Catal. ACS Cent. Sci. ACS Chem. Biol. ACS Chemical Health & Safety ACS Chem. Neurosci. ACS Comb. Sci. ACS Earth Space Chem. ACS Energy Lett. ACS Infect. Dis. ACS Macro Lett. ACS Mater. Lett. ACS Med. Chem. Lett. ACS Nano ACS Omega ACS Photonics ACS Sens. ACS Sustainable Chem. Eng. ACS Synth. Biol. Anal. Chem. BIOCHEMISTRY-US Bioconjugate Chem. BIOMACROMOLECULES Chem. Res. Toxicol. Chem. Rev. Chem. Mater. CRYST GROWTH DES ENERG FUEL Environ. Sci. Technol. Environ. Sci. Technol. Lett. Eur. J. Inorg. Chem. IND ENG CHEM RES Inorg. Chem. J. Agric. Food. Chem. J. Chem. Eng. Data J. Chem. Educ. J. Chem. Inf. Model. J. Chem. Theory Comput. J. Med. Chem. J. Nat. Prod. J PROTEOME RES J. Am. Chem. Soc. LANGMUIR MACROMOLECULES Mol. Pharmaceutics Nano Lett. Org. Lett. ORG PROCESS RES DEV ORGANOMETALLICS J. Org. Chem. J. Phys. Chem. J. Phys. Chem. A J. Phys. Chem. B J. Phys. Chem. C J. Phys. Chem. Lett. Analyst Anal. Methods Biomater. Sci. Catal. Sci. Technol. Chem. Commun. Chem. Soc. Rev. CHEM EDUC RES PRACT CRYSTENGCOMM Dalton Trans. Energy Environ. Sci. ENVIRON SCI-NANO ENVIRON SCI-PROC IMP ENVIRON SCI-WAT RES Faraday Discuss. Food Funct. Green Chem. Inorg. Chem. Front. Integr. Biol. J. Anal. At. Spectrom. J. Mater. Chem. A J. Mater. Chem. B J. Mater. Chem. C Lab Chip Mater. Chem. Front. Mater. Horiz. MEDCHEMCOMM Metallomics Mol. Biosyst. Mol. Syst. Des. Eng. Nanoscale Nanoscale Horiz. Nat. Prod. Rep. New J. Chem. Org. Biomol. Chem. Org. Chem. Front. PHOTOCH PHOTOBIO SCI PCCP Polym. Chem.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1