{"title":"Using single impact metrics to assess research in business and economics: why institutions should use multi-criteria systems for assessing research","authors":"S. Olavarrieta","doi":"10.1108/jefas-04-2021-0033","DOIUrl":null,"url":null,"abstract":"PurposeDespite the general recommendation of using a combination of multiple criteria for research assessment and faculty promotion decisions, the raise of quantitative indicators is generating an emerging trend in Business Schools to use single journal impact factors (IFs) as key (unique) drivers for those relevant school decisions. This paper aims to investigate the effects of using single Web of Science (WoS)-based journal impact metrics when assessing research from two related disciplines: Business and Economics, and its potential impact for the strategic sustainability of a Business School.Design/methodology/approachThis study collected impact indicators data for Business and Economics journals from the Clarivate Web of Science database. We concentrated on the IF indicators, the Eigenfactor and the article influence score (AIS). This study examined the correlations between these indicators and then ranked disciplines and journals using these different impact metrics.FindingsConsistent with previous findings, this study finds positive correlations among these metrics. Then this study ranks the disciplines and journals using each impact metric, finding relevant and substantial differences, depending on the metric used. It is found that using AIS instead of the IF raises the relative ranking of Economics, while Business remains basically with the same rank.Research limitations/implications This study contributes to the research assessment literature by adding substantial evidence that given the sensitivity of journal rankings to particular indicators, the selection of a single impact metric for assessing research and hiring/promotion and tenure decisions is risky and too simplistic. This research shows that biases may be larger when assessment involves researchers from related disciplines – like Business and Economics – but with different research foundations and traditions.Practical implicationsConsistent with the literature, given the sensibility of journal rankings to particular indicators, the selection of a single impact metric for assessing research, assigning research funds and hiring/promotion and tenure decisions is risky and simplistic. However, this research shows that risks and biases may be larger when assessment involves researchers from related disciplines – like Business and Economics – but with different research foundations and trajectories. The use of multiple criteria is advised for such purposes.Originality/valueThis is an applied work using real data from WoS that addresses a practical case of comparing the use of different journal IFs to rank-related disciplines like Business and Economics, with important implications for faculty tenure and promotion committees and for research funds granting institutions and decision-makers.","PeriodicalId":53491,"journal":{"name":"Journal of Economics, Finance and Administrative Science","volume":"26 1","pages":""},"PeriodicalIF":2.3000,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"2","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Economics, Finance and Administrative Science","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1108/jefas-04-2021-0033","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ECONOMICS","Score":null,"Total":0}
引用次数: 2
Abstract
PurposeDespite the general recommendation of using a combination of multiple criteria for research assessment and faculty promotion decisions, the raise of quantitative indicators is generating an emerging trend in Business Schools to use single journal impact factors (IFs) as key (unique) drivers for those relevant school decisions. This paper aims to investigate the effects of using single Web of Science (WoS)-based journal impact metrics when assessing research from two related disciplines: Business and Economics, and its potential impact for the strategic sustainability of a Business School.Design/methodology/approachThis study collected impact indicators data for Business and Economics journals from the Clarivate Web of Science database. We concentrated on the IF indicators, the Eigenfactor and the article influence score (AIS). This study examined the correlations between these indicators and then ranked disciplines and journals using these different impact metrics.FindingsConsistent with previous findings, this study finds positive correlations among these metrics. Then this study ranks the disciplines and journals using each impact metric, finding relevant and substantial differences, depending on the metric used. It is found that using AIS instead of the IF raises the relative ranking of Economics, while Business remains basically with the same rank.Research limitations/implications This study contributes to the research assessment literature by adding substantial evidence that given the sensitivity of journal rankings to particular indicators, the selection of a single impact metric for assessing research and hiring/promotion and tenure decisions is risky and too simplistic. This research shows that biases may be larger when assessment involves researchers from related disciplines – like Business and Economics – but with different research foundations and traditions.Practical implicationsConsistent with the literature, given the sensibility of journal rankings to particular indicators, the selection of a single impact metric for assessing research, assigning research funds and hiring/promotion and tenure decisions is risky and simplistic. However, this research shows that risks and biases may be larger when assessment involves researchers from related disciplines – like Business and Economics – but with different research foundations and trajectories. The use of multiple criteria is advised for such purposes.Originality/valueThis is an applied work using real data from WoS that addresses a practical case of comparing the use of different journal IFs to rank-related disciplines like Business and Economics, with important implications for faculty tenure and promotion committees and for research funds granting institutions and decision-makers.
尽管普遍建议在研究评估和教师晋升决策中结合使用多种标准,但定量指标的增加正在商学院中产生一种新趋势,即使用单一期刊影响因子(if)作为相关学院决策的关键(独特)驱动因素。本文旨在探讨使用基于Web of Science (WoS)的期刊影响指标评估商业和经济学两个相关学科研究的效果,以及其对商学院战略可持续性的潜在影响。设计/方法/方法本研究从Clarivate Web of Science数据库中收集商业和经济期刊的影响指标数据。我们重点研究了影响因子指标、特征因子和文章影响评分(AIS)。本研究检查了这些指标之间的相关性,然后使用这些不同的影响指标对学科和期刊进行排名。研究结果与先前的研究结果一致,本研究发现这些指标之间存在正相关。然后,本研究使用每个影响指标对学科和期刊进行排名,根据所使用的指标找到相关的和实质性的差异。结果发现,用AIS代替IF提高了经济学的相对排名,而商科的排名基本保持不变。本研究为研究评估文献做出了贡献,增加了大量证据,表明鉴于期刊排名对特定指标的敏感性,选择单一的影响指标来评估研究、招聘/晋升和终身教职决策是有风险的,而且过于简单。这项研究表明,当评估涉及来自相关学科(如商业和经济学)但研究基础和传统不同的研究人员时,偏见可能会更大。实际意义与文献一致,考虑到期刊排名对特定指标的敏感性,选择单一的影响指标来评估研究、分配研究经费、雇用/晋升和终身教职决策是有风险和简单的。然而,这项研究表明,当评估涉及来自相关学科(如商业和经济学)但具有不同研究基础和轨迹的研究人员时,风险和偏差可能更大。为此,建议使用多个标准。原创性/价值这是一项应用工作,使用WoS的真实数据,解决了一个实际案例,比较了不同期刊IFs与排名相关学科(如商业和经济学)的使用情况,对教师终身教职和晋升委员会以及研究基金授予机构和决策者具有重要意义。
期刊介绍:
The Universidad ESAN, with more than 50 years of experience in the higher education field and post graduate studies, desires to contribute to the academic community with the most outstanding pieces of research. We gratefully welcome suggestions and contributions from business areas such as operations, supply chain, economics, finance and administration. We publish twice a year, six articles for each issue.