The usefulness of personal publication lists in research evaluation

Abstract

This article addresses the question of whether personal publication lists should be used as a data source in research evaluation, or whether, as is widespread in practice, existing databases, such as Web of Science, can be used instead. For this purpose, an empirical study was carried out in which all business administration university professors (n = 233) of a non-English-speaking country, namely Austria, were ranked in several ways (e.g., full or fractional counting, consideration or non-consideration of journal rankings). All rankings were based on the number of published journal articles (n = 4246; observation period: 10 years). In one case, the personal publication lists and in the other case, the Web of Science were used as data source for these rankings. The rankings created in these two ways were compared with each other. The results show that the choice of the data source has a major influence on the ranking results. For researchers from non-English-speaking countries with (many) publications in their respective national languages, an exclusive use of international databases, such as Web of Science in our case, cannot fully consider the whole research performance. In these cases, the use of personal publication lists seems to make a lot of sense, at least for several ranking variants, despite the effort involved. The main contribution of our study is that we compare personal publication lists as a data source with Web of Science which is often used in research evaluations. In addition, this comparison is not, as usual, input-related (based on the degree of coverage in the two data sources) but impact-related (based on rankings that are created based on the publications contained in the two data sources).