Librarian and Researcher Assessments of Search Result Relevance: How Well Do They Align?

Abstract

Conducting comprehensive but efficient literature searches for complex evidence syntheses involves selecting databases that will retrieve the greatest number of relevant results on the question. Lack of a comprehensive single database on allied health educational topics challenges those seeking such literature. In this study, six participants contributed research questions on instructional methods and materials for allied health patients, caregivers, and future health professionals. Two health sciences librarians created search strategies for these questions and searched eleven databases. Both the librarians and the six participants evaluated the search results using a rubric based on PICO to assess extent of alignment between the librarians' and requestors' relevance judgments. Intervention, Outcome, and Assessment Method constituted the most frequent bases for assessments of relevance by both librarians and participants. The librarians were more restrictive in all of their assessments except in a preliminary search yielding twelve citations without abstracts. The study's results could be used to identify effective techniques for reference interviewing, selecting databases, and weeding search results.