Developing a comprehensive evaluation questionnaire for university FAQ administration chatbots

Abstract

A chatbot is a domain-specific conversational interface that mimics human assistance for users of various systems. Recently chatbots have received much research interest in supporting university administrative operations. However, rapid and large-scale implementation of chatbots in university administration systems remains challenged. Extant literature reflects on this challenge from various perspectives including, technical, managerial and socio-technical lenses. This paper heralds a somewhat overlooked perspective namely, the processes and techniques for concise and rigorous evaluation of these chatbots. The distinctiveness of this paper lies in the tri-perspectives of anthropomorphism, usability and user experience which converge to provide a stronger lens for chatbot evaluation particularly in a university administration setting. Recent studies primarily devise heuristic methods that tend to evaluate chatbots in silos such as, user interface, usability or the conversation ability and quality. There is a noticeable lack of research that attempts combination of these seemingly complex areas of chatbot evaluation. This paper postulates improved rigour of evaluation if coverage is expanded to usability, anthropomorphism, acceptance, usage, and user interface. The aim of this paper is therefore, to design a novel evaluation instrument tailored for a university administration chatbot. This is achieved by implementing the well-known Unified Technology Acceptance and Use of Technology framework as the architectural underpinning. Constituent components of the instrument derive from recent literature and emerging trends in frequently asked questions-based chatbot evaluation. The major contribution stems from the identification and insertion of key and overlooked evaluation perspectives which culminate in a more rigorous and a more encompassing evaluation questionnaire.