Comment-enriched index terms improve the relevance and novelty of the ranking of the commented medical articles retrieved by an NLP system

Abstract

PurposeThis study aimed to investigate how the enrichment of medical documents' index terms by their comments improves the relevance and novelty of the top-ranked results retrieved by an NLP system.Design/methodology/approachA semi-experimental pre-test and post-test research was designed to compare NLP-based indexes before and after being expanded by the comment terms. The experiments were conducted on a test collection of 13,957 documents commented by F1000-Prime reviewers. They were indexed at title, abstract, body and full-text levels. In total, 100 seed documents were randomly selected and served as queries. The textual similarity of the documents and queries was calculated using Lucene-more-like-this function and evaluated by the semantic similarity of their MeSH. The results novelty was measured using maximal marginal relevance and evaluated by their MeSH novelties. Normalized discounted cumulative gain was used to compare the basic and expanded indexes' precisions at 10, 20 and 50 top ranks.FindingsThe relevance and novelty of the results ranked at the top precision points was improved after expanding the indexes by the comment terms. The finding implies that meta-texts are effective in representing their mother documents, by adding dynamic elements to their rather static contents. It also provides further evidence about the merits of the application of social intelligence and collective wisdom reflected in the actions and reactions of users in tackling the challenges faced by NLP-based systems.Originality/valueThis is the first study to confirm that social comments on scientific papers improve the performance of information systems in terms of relevance and novelty.Peer reviewThe peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-05-2022-0283.