Kirsi Pulkkinen, Timo Aarrevaara, Mikko Rask, Markku Mattila
In this paper we investigate the practices and capacities that define successful societal interaction of research groups with stakeholders in mutually beneficial processes. We studied the Finnish Strategic Research Council’s (SRC) first funded projects through a dynamic governance lens. The aim of the paper is to explore how the societal interaction was designed and commenced at the onset of the projects in order to understand the logic through which the consortia expected broad impacts to occur. The Finnish SRC introduced a societal interaction plan (SIP) approach, which requires research consortia to consider societal interaction alongside research activities in a way that exceeds conventional research plans. Hence, the first SRC projects’ SIPs and the implemented activities and working logics discussed in the interviews provide a window into exploring how active societal interaction reflects the call for dynamic, sustainable practices and new capabilities to better link research to societal development. We found that the capacities of dynamic governance were implemented by integrating societal interaction into research, in particular through a ‘drizzling’ approach. In these emerging practices SIP designs function as platforms for the formation of communities of experts, rather than traditional project management models or mere communication tools. The research groups utilized the benefits of pooling academic knowledge and skills with other types of expertise for mutual gain. They embraced the limits of expertise and reached out to societal partners to truly broker knowledge, and exchange and develop capacities and perspectives to solve grand societal challenges.
{"title":"Societal interaction plans—A tool for enhancing societal engagement of strategic research in Finland","authors":"Kirsi Pulkkinen, Timo Aarrevaara, Mikko Rask, Markku Mattila","doi":"10.1093/reseval/rvae002","DOIUrl":"https://doi.org/10.1093/reseval/rvae002","url":null,"abstract":"\u0000 In this paper we investigate the practices and capacities that define successful societal interaction of research groups with stakeholders in mutually beneficial processes. We studied the Finnish Strategic Research Council’s (SRC) first funded projects through a dynamic governance lens. The aim of the paper is to explore how the societal interaction was designed and commenced at the onset of the projects in order to understand the logic through which the consortia expected broad impacts to occur. The Finnish SRC introduced a societal interaction plan (SIP) approach, which requires research consortia to consider societal interaction alongside research activities in a way that exceeds conventional research plans. Hence, the first SRC projects’ SIPs and the implemented activities and working logics discussed in the interviews provide a window into exploring how active societal interaction reflects the call for dynamic, sustainable practices and new capabilities to better link research to societal development. We found that the capacities of dynamic governance were implemented by integrating societal interaction into research, in particular through a ‘drizzling’ approach. In these emerging practices SIP designs function as platforms for the formation of communities of experts, rather than traditional project management models or mere communication tools. The research groups utilized the benefits of pooling academic knowledge and skills with other types of expertise for mutual gain. They embraced the limits of expertise and reached out to societal partners to truly broker knowledge, and exchange and develop capacities and perspectives to solve grand societal challenges.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":null,"pages":null},"PeriodicalIF":3.3,"publicationDate":"2024-06-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141365480","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Prompted by research funding agencies and regulatory bodies, universities need to demonstrate and manage the impact of their research. Therefore, models for evaluating research impacts that can be applied quickly and at a reasonable cost are needed. However, the development of these models presents considerable challenges: research impact is multidimensional and can occur over time because of multiple interactions between different agents. Furthermore, impact assessment can demand substantial time and resources. Considering this challenging context, this study proposes a research impact assessment model capable of optimizing benefits and costs. The model was based on a multidimensional impact perspective and an application-oriented way of generating and transmitting knowledge. It was applied to a large teaching and research institution with a predominant role in the social sciences. The assessment involved a survey of the leaders of 133 research projects. The main findings of the application of the assessment model were as follows: first, knowledge production is transitioning from a traditional mode (aimed to impact science) to a new mode (aimed to impact multiple stakeholders); second, project leaders still perceive the prevalence of scientific impact over other types of impact; and third, the survey revealed how certain characteristics of the knowledge production mode relate to (perceived) impact. The viable research impact assessment model was demonstrated to be useful and cost-effective; thus, with adaptations and improvements, it may be used by other institutions, particularly those with internal research funding programs. This article also presents the model’s limitations, suggesting directions for future research.
{"title":"Research impact assessment: Developing and applying a viable model for the social sciences","authors":"Thomaz Wood, Adriana Wilner","doi":"10.1093/reseval/rvae022","DOIUrl":"https://doi.org/10.1093/reseval/rvae022","url":null,"abstract":"\u0000 Prompted by research funding agencies and regulatory bodies, universities need to demonstrate and manage the impact of their research. Therefore, models for evaluating research impacts that can be applied quickly and at a reasonable cost are needed. However, the development of these models presents considerable challenges: research impact is multidimensional and can occur over time because of multiple interactions between different agents. Furthermore, impact assessment can demand substantial time and resources. Considering this challenging context, this study proposes a research impact assessment model capable of optimizing benefits and costs. The model was based on a multidimensional impact perspective and an application-oriented way of generating and transmitting knowledge. It was applied to a large teaching and research institution with a predominant role in the social sciences. The assessment involved a survey of the leaders of 133 research projects. The main findings of the application of the assessment model were as follows: first, knowledge production is transitioning from a traditional mode (aimed to impact science) to a new mode (aimed to impact multiple stakeholders); second, project leaders still perceive the prevalence of scientific impact over other types of impact; and third, the survey revealed how certain characteristics of the knowledge production mode relate to (perceived) impact. The viable research impact assessment model was demonstrated to be useful and cost-effective; thus, with adaptations and improvements, it may be used by other institutions, particularly those with internal research funding programs. This article also presents the model’s limitations, suggesting directions for future research.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":null,"pages":null},"PeriodicalIF":3.3,"publicationDate":"2024-05-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141126086","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Endorsed by the European Research Area, a Coalition for Advancing Research Assessment (CoARA), primarily composed of research institutions and funders, was established in 2022. Its mission is to reform research assessment, emphasizing a shift towards qualitative judgment. This study examines the CoARA initiative from theoretical, practical, and applicative perspectives, exploring its implications. The manuscript questions a standardized way to evaluate research suitable in every context. Through empirical evidence and internal logical arguments, it challenges the supremacy of qualitative over quantitative methods in terms of robustness, accuracy, validity, functionality, and time and cost convenience in STEMM research assessment. The aim is to clear up common misunderstandings and offer insights to policymakers and decision-makers in charge of defining how research should be evaluated.
{"title":"The forced battle between peer-review and scientometric research assessment: Why the CoARA initiative is unsound","authors":"Giovanni Abramo","doi":"10.1093/reseval/rvae021","DOIUrl":"https://doi.org/10.1093/reseval/rvae021","url":null,"abstract":"\u0000 Endorsed by the European Research Area, a Coalition for Advancing Research Assessment (CoARA), primarily composed of research institutions and funders, was established in 2022. Its mission is to reform research assessment, emphasizing a shift towards qualitative judgment. This study examines the CoARA initiative from theoretical, practical, and applicative perspectives, exploring its implications. The manuscript questions a standardized way to evaluate research suitable in every context. Through empirical evidence and internal logical arguments, it challenges the supremacy of qualitative over quantitative methods in terms of robustness, accuracy, validity, functionality, and time and cost convenience in STEMM research assessment. The aim is to clear up common misunderstandings and offer insights to policymakers and decision-makers in charge of defining how research should be evaluated.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":null,"pages":null},"PeriodicalIF":3.3,"publicationDate":"2024-05-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140988461","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
There is a robust literature documenting differences in peer review processes for scholarly outlets. Knowledge of this variability has provoked thoughtful debate about the best approach for promoting rigor and innovation in scientific research (e.g. single-blind vs. double-blind review, or more recently, double-blind vs. open review). We aim to expand this conversation to external peer review processes specified in rank and tenure guidelines. We qualitatively analyze a corpus of publicly available rank and tenure procedures at research-intensive universities in the United States. Results indicate significant variation in (1) the required minimum and maximum number of external reviews, (2) the candidate’s role in the reviewer selection process, (3) the level of ensured anonymity for reviewers, and (4) attention to potential conflict of interest scenarios. We argue that many of the debates about best practices in research evaluation for journals and funding agencies are also relevant for rank and tenure procedures. Moreover, rank and tenure policies may be subject to unique tensions, such as cases where external reviews are formally cast as disinterested assessments by referees but informally understood as letters of recommendation by sponsors. We discuss the relevance of our findings for existing work on the perceived clarity of evaluation criteria—especially how guideline ambiguity can be linked to inequality and how labor issues can conflict with idealized assessment principles.
{"title":"Policy variation in the external evaluation of research for tenure at U.S. universities","authors":"Lance Hannon, Meredith Bergey","doi":"10.1093/reseval/rvae018","DOIUrl":"https://doi.org/10.1093/reseval/rvae018","url":null,"abstract":"\u0000 There is a robust literature documenting differences in peer review processes for scholarly outlets. Knowledge of this variability has provoked thoughtful debate about the best approach for promoting rigor and innovation in scientific research (e.g. single-blind vs. double-blind review, or more recently, double-blind vs. open review). We aim to expand this conversation to external peer review processes specified in rank and tenure guidelines. We qualitatively analyze a corpus of publicly available rank and tenure procedures at research-intensive universities in the United States. Results indicate significant variation in (1) the required minimum and maximum number of external reviews, (2) the candidate’s role in the reviewer selection process, (3) the level of ensured anonymity for reviewers, and (4) attention to potential conflict of interest scenarios. We argue that many of the debates about best practices in research evaluation for journals and funding agencies are also relevant for rank and tenure procedures. Moreover, rank and tenure policies may be subject to unique tensions, such as cases where external reviews are formally cast as disinterested assessments by referees but informally understood as letters of recommendation by sponsors. We discuss the relevance of our findings for existing work on the perceived clarity of evaluation criteria—especially how guideline ambiguity can be linked to inequality and how labor issues can conflict with idealized assessment principles.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":null,"pages":null},"PeriodicalIF":3.3,"publicationDate":"2024-05-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141005993","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Anders Hylmö, Kody Steffy, Duncan A Thomas, Liv Langfeldt
Whereas a growing number of studies evidence that research quality notions and evaluative practices are field- and context-specific, many focus on single evaluative practices or moments. This paper introduces the concept of quality landscape to capture dynamics of interrelated quality notions, evaluative moments and practices in a research field. This concept shifts focus to (1) the field-specific universe of practices, devices and notions of research quality; (2) ways that interrelated valuations provide structure and boundeness to a landscape; (3) ways that perspectives on a shared landscape may change with position within the landscape; and (4) ways in which a quality landscape is intertwined with the field’s socio-epistemic conditions. With extensive interview data from top ranked departments in three Scandinavian countries, we use economics as a case for exploring the value of a quality landscape lens. We find that the field’s journal hierarchy and its ‘Top 5’ journals dominate the landscape, while other important evaluative practices beyond the top five are interlinked with the journal hierarchy. However, quantitative evaluative metrics common in other fields are virtually absent. We further find that national and local policy reinforce the journal hierarchy emphasis, and that career stages affect quality perspectives. We argue that the quality landscape is structured as a quality hierarchy with a focus on the core ‘general interest’, and suggest the notion of ordinalization (the process of rank ordering) as an organizing principle linking the quality landscape to the field’s socio-epistemic conditions. Finally, we offer suggestions for further research.
{"title":"The quality landscape of economics: The top five and beyond","authors":"Anders Hylmö, Kody Steffy, Duncan A Thomas, Liv Langfeldt","doi":"10.1093/reseval/rvae014","DOIUrl":"https://doi.org/10.1093/reseval/rvae014","url":null,"abstract":"\u0000 Whereas a growing number of studies evidence that research quality notions and evaluative practices are field- and context-specific, many focus on single evaluative practices or moments. This paper introduces the concept of quality landscape to capture dynamics of interrelated quality notions, evaluative moments and practices in a research field. This concept shifts focus to (1) the field-specific universe of practices, devices and notions of research quality; (2) ways that interrelated valuations provide structure and boundeness to a landscape; (3) ways that perspectives on a shared landscape may change with position within the landscape; and (4) ways in which a quality landscape is intertwined with the field’s socio-epistemic conditions. With extensive interview data from top ranked departments in three Scandinavian countries, we use economics as a case for exploring the value of a quality landscape lens. We find that the field’s journal hierarchy and its ‘Top 5’ journals dominate the landscape, while other important evaluative practices beyond the top five are interlinked with the journal hierarchy. However, quantitative evaluative metrics common in other fields are virtually absent. We further find that national and local policy reinforce the journal hierarchy emphasis, and that career stages affect quality perspectives. We argue that the quality landscape is structured as a quality hierarchy with a focus on the core ‘general interest’, and suggest the notion of ordinalization (the process of rank ordering) as an organizing principle linking the quality landscape to the field’s socio-epistemic conditions. Finally, we offer suggestions for further research.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":null,"pages":null},"PeriodicalIF":3.3,"publicationDate":"2024-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140659141","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Research agencies increasingly use thematic funding schemes to steer scientists toward issues of political interest. Funders set strategic priorities and expect scientists to follow the money. Using statistical-matching methods, we paired 10,475 UK-based recipients of thematic grants (with aims proposed by the funding agency) and responsive-mode grants (with aims proposed by researchers) in Physics, Engineering, and Bioscience to investigate the relation between funding-type and shifts in scientists’ research interests. We applied citation-based community detection to create individual time-series of topic distributions based on the grantees’ publications, and used these to estimate funding-related shifts in research focus. Our analysis, which estimates the similarity of the grantees’ research focus before, during and after a grant, suggests that scientists acquiring thematic funding alter their research interests more than comparable scientists funded through responsive-mode schemes. However, recipients of both types of grants tend to revert toward their original research interests when the funding expires. We find no evidence that thematic funding increases the level of diversity in a recipient’s research portfolio. Our study thus highlights an ambiguous link between thematic funding mechanisms and researchers’ orientations.
{"title":"Do thematic funding instruments lead researchers in new directions? Strategic funding priorities and topic switching among British grant recipients","authors":"Emil Bargmann Madsen, Mathias Wullum Nielsen","doi":"10.1093/reseval/rvae015","DOIUrl":"https://doi.org/10.1093/reseval/rvae015","url":null,"abstract":"\u0000 Research agencies increasingly use thematic funding schemes to steer scientists toward issues of political interest. Funders set strategic priorities and expect scientists to follow the money. Using statistical-matching methods, we paired 10,475 UK-based recipients of thematic grants (with aims proposed by the funding agency) and responsive-mode grants (with aims proposed by researchers) in Physics, Engineering, and Bioscience to investigate the relation between funding-type and shifts in scientists’ research interests. We applied citation-based community detection to create individual time-series of topic distributions based on the grantees’ publications, and used these to estimate funding-related shifts in research focus. Our analysis, which estimates the similarity of the grantees’ research focus before, during and after a grant, suggests that scientists acquiring thematic funding alter their research interests more than comparable scientists funded through responsive-mode schemes. However, recipients of both types of grants tend to revert toward their original research interests when the funding expires. We find no evidence that thematic funding increases the level of diversity in a recipient’s research portfolio. Our study thus highlights an ambiguous link between thematic funding mechanisms and researchers’ orientations.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":null,"pages":null},"PeriodicalIF":3.3,"publicationDate":"2024-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140665784","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Daniel P Wadsworth, Alison Craswell, Andy Ward, Kristen Tulloch, Mia A Schaumberg, Lee Stoner, Pamela J. Meredith, Christopher D. Askew
The academic world is rapidly changing due to multiple influences, including COVID-19, technological advancement, tertiary competition, shifting government policies, and emerging research practices and priorities. Given this rapidly changing tertiary-education landscape, added importance is placed on the need for emerging research leaders to understand contemporary research practices which likely exceeds the skills displayed by their supervisors and mentors. Consequently, reflecting on 80+ years of collective post-PhD research experience, we identify and discuss six common characteristics of contemporary health research practice, with the aim of guiding and empowering higher degree by research students and emerging early career researchers as they develop their research identities and shape their approaches and values on the path to becoming research leaders in health. Indicative of today’s dynamic and exacting research environment, we propose that best practice contemporary health research should be: Diverse; Creative; Collaborative; Authentic; Respectful; and, Trusting. By understanding a diverse range of research approaches, and through working in collaborative and creative teams, diversity of thought, approach, and methods can become a cornerstone of practice. By blending this understanding with questions and research approaches that facilitate the pathway of translation uptake for the next-user, emerging research leaders can deliver authentic research with the requisite ingredients for impact.
{"title":"Characteristics of contemporary health research practice: A shift from ivory tower to collaborative power","authors":"Daniel P Wadsworth, Alison Craswell, Andy Ward, Kristen Tulloch, Mia A Schaumberg, Lee Stoner, Pamela J. Meredith, Christopher D. Askew","doi":"10.1093/reseval/rvae006","DOIUrl":"https://doi.org/10.1093/reseval/rvae006","url":null,"abstract":"\u0000 The academic world is rapidly changing due to multiple influences, including COVID-19, technological advancement, tertiary competition, shifting government policies, and emerging research practices and priorities. Given this rapidly changing tertiary-education landscape, added importance is placed on the need for emerging research leaders to understand contemporary research practices which likely exceeds the skills displayed by their supervisors and mentors. Consequently, reflecting on 80+ years of collective post-PhD research experience, we identify and discuss six common characteristics of contemporary health research practice, with the aim of guiding and empowering higher degree by research students and emerging early career researchers as they develop their research identities and shape their approaches and values on the path to becoming research leaders in health. Indicative of today’s dynamic and exacting research environment, we propose that best practice contemporary health research should be: Diverse; Creative; Collaborative; Authentic; Respectful; and, Trusting. By understanding a diverse range of research approaches, and through working in collaborative and creative teams, diversity of thought, approach, and methods can become a cornerstone of practice. By blending this understanding with questions and research approaches that facilitate the pathway of translation uptake for the next-user, emerging research leaders can deliver authentic research with the requisite ingredients for impact.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":null,"pages":null},"PeriodicalIF":3.3,"publicationDate":"2024-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140720942","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
David Johann, Jörg Neufeld, Kathrin Thomas, Justus M. K. Rathmann, Heiko Rauhut
This article investigates researchers’ publication strategies and how their perceived pressure to publish and to obtain external funding are related to these strategies. The analyses rely on data from the Zurich Survey of Academics (ZSoA), an online survey representative of academics working at higher education institutions in Germany, Austria, and Switzerland. The results suggest that academics pursue both instrumental and normative publication strategies. The main finding is that academics who perceive high pressure to publish tend to employ instrumental publication strategies rather than normative ones: they are more likely to focus on the journal's reputation and the speed of publication when selecting an outlet for peer review. Publishing results in open-access outlets or in native languages other than English is less important for those under pressure. However, the extent to which researchers’ perceived pressure affects publication strategies also depends on other factors, such as the discrepancy between the time available for research and the time actually desired for research.
{"title":"The impact of researchers’ perceived pressure on their publication strategies","authors":"David Johann, Jörg Neufeld, Kathrin Thomas, Justus M. K. Rathmann, Heiko Rauhut","doi":"10.1093/reseval/rvae011","DOIUrl":"https://doi.org/10.1093/reseval/rvae011","url":null,"abstract":"\u0000 This article investigates researchers’ publication strategies and how their perceived pressure to publish and to obtain external funding are related to these strategies. The analyses rely on data from the Zurich Survey of Academics (ZSoA), an online survey representative of academics working at higher education institutions in Germany, Austria, and Switzerland. The results suggest that academics pursue both instrumental and normative publication strategies. The main finding is that academics who perceive high pressure to publish tend to employ instrumental publication strategies rather than normative ones: they are more likely to focus on the journal's reputation and the speed of publication when selecting an outlet for peer review. Publishing results in open-access outlets or in native languages other than English is less important for those under pressure. However, the extent to which researchers’ perceived pressure affects publication strategies also depends on other factors, such as the discrepancy between the time available for research and the time actually desired for research.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":null,"pages":null},"PeriodicalIF":3.3,"publicationDate":"2024-03-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140384502","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Recent times have seen the growth in the number and scope of interacting professional reform movements in science, centered on themes such as open research, research integrity, responsible research assessment, and responsible metrics. The responsible metrics movement identifies the growing influence of quantitative performance indicators as a major problem and seeks to steer and improve practices around their use. It is a multi-actor, multi-disciplinary reform movement premised upon engendering a sense of responsibility among academic evaluators to approach metrics with caution and avoid certain poor practices. In this article we identify how academic evaluators engage with the responsible metrics agenda, via semi-structured interview and open-text survey responses on professorial hiring, tenure and promotion assessments among senior academics in the United States—a country that has so far been less visibly engaged with the responsible metrics reform agenda. We explore how notions of ‘responsibility’ are experienced and practiced among the very types of professionals international reform initiatives such as the San Francisco Declaration on Research Assessment (DORA) are hoping to mobilize into their cause. In doing so, we draw on concepts from science studies, including from literatures on Responsible Research and Innovation and ‘folk theories’ of citation. We argue that literature on citation folk theories should extend its scope beyond simply asking researchers how they view the role and validity of these tools as performance measures, by asking them also what they consider are their professional obligations to handle bibliometrics appropriately.
{"title":"Practicing responsible research assessment: Qualitative study of faculty hiring, promotion, and tenure assessments in the United States","authors":"Alex Rushforth, Sarah de Rijcke","doi":"10.1093/reseval/rvae007","DOIUrl":"https://doi.org/10.1093/reseval/rvae007","url":null,"abstract":"\u0000 Recent times have seen the growth in the number and scope of interacting professional reform movements in science, centered on themes such as open research, research integrity, responsible research assessment, and responsible metrics. The responsible metrics movement identifies the growing influence of quantitative performance indicators as a major problem and seeks to steer and improve practices around their use. It is a multi-actor, multi-disciplinary reform movement premised upon engendering a sense of responsibility among academic evaluators to approach metrics with caution and avoid certain poor practices. In this article we identify how academic evaluators engage with the responsible metrics agenda, via semi-structured interview and open-text survey responses on professorial hiring, tenure and promotion assessments among senior academics in the United States—a country that has so far been less visibly engaged with the responsible metrics reform agenda. We explore how notions of ‘responsibility’ are experienced and practiced among the very types of professionals international reform initiatives such as the San Francisco Declaration on Research Assessment (DORA) are hoping to mobilize into their cause. In doing so, we draw on concepts from science studies, including from literatures on Responsible Research and Innovation and ‘folk theories’ of citation. We argue that literature on citation folk theories should extend its scope beyond simply asking researchers how they view the role and validity of these tools as performance measures, by asking them also what they consider are their professional obligations to handle bibliometrics appropriately.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":null,"pages":null},"PeriodicalIF":3.3,"publicationDate":"2024-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140244723","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
José Luis Jiménez-Andrade, R. Arencibia-Jorge, Miguel Robles-Pérez, Julia Tagüeña, T. Govezensky, Humberto Carrillo-Calvet, Rafael A Barrio, K. Kaski
This paper analyzes the research performance evolution of a scientific institute, from its genesis through various stages of development. The main aim is to obtain, and visually represent, bibliometric evidence of the correlation of organizational changes on the development of its scientific performance; particularly, structural and leadership changes. The study involves six bibliometric indicators to multidimensionally assess the evolution of the institution’s performance profile. For a case study, we selected the Renewable Energy Institute at the National Autonomous University of Mexico, created 35 years ago as a small laboratory, then it evolved to a research center and finally to a formal institute, which over the last 8 years changed from the traditional departmental structure to a network-based structure. The evolution of the multidimensional performance profiles is analyzed, and graphically represented, using a novel artificial intelligence-based approach. We analyzed the performance profiles evolution yearly, using Principal Components Analysis, and a self-organizing neural network mapping technique. This approach, combining bibliometric and machine learning techniques, proved to be effective for the assessment of the institution’s evolution process. The results were represented with a series of graphs and maps that clearly reveal the magnitude and nature of the performance profile evolution, as well as its correlation with each of the structural and leadership transitions. These exploratory results have provided us data and insights into the probable effects of these transitions on academic performance, that have been useful to create a dynamical model.
{"title":"Organizational changes and research performance: A multidimensional assessment","authors":"José Luis Jiménez-Andrade, R. Arencibia-Jorge, Miguel Robles-Pérez, Julia Tagüeña, T. Govezensky, Humberto Carrillo-Calvet, Rafael A Barrio, K. Kaski","doi":"10.1093/reseval/rvae005","DOIUrl":"https://doi.org/10.1093/reseval/rvae005","url":null,"abstract":"\u0000 This paper analyzes the research performance evolution of a scientific institute, from its genesis through various stages of development. The main aim is to obtain, and visually represent, bibliometric evidence of the correlation of organizational changes on the development of its scientific performance; particularly, structural and leadership changes. The study involves six bibliometric indicators to multidimensionally assess the evolution of the institution’s performance profile. For a case study, we selected the Renewable Energy Institute at the National Autonomous University of Mexico, created 35 years ago as a small laboratory, then it evolved to a research center and finally to a formal institute, which over the last 8 years changed from the traditional departmental structure to a network-based structure. The evolution of the multidimensional performance profiles is analyzed, and graphically represented, using a novel artificial intelligence-based approach. We analyzed the performance profiles evolution yearly, using Principal Components Analysis, and a self-organizing neural network mapping technique. This approach, combining bibliometric and machine learning techniques, proved to be effective for the assessment of the institution’s evolution process. The results were represented with a series of graphs and maps that clearly reveal the magnitude and nature of the performance profile evolution, as well as its correlation with each of the structural and leadership transitions. These exploratory results have provided us data and insights into the probable effects of these transitions on academic performance, that have been useful to create a dynamical model.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":null,"pages":null},"PeriodicalIF":3.3,"publicationDate":"2024-02-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139963169","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：