An LLM-Driven Chatbot in Higher Education for Databases and Information Systems

Abstract

Contribution: This research explores the benefits and challenges of developing, deploying, and evaluating a large language model (LLM) chatbot, MoodleBot, in computer science classroom settings. It highlights the potential of integrating LLMs into LMSs like Moodle to support self-regulated learning (SRL) and help-seeking behavior. Background: Computer science educators face immense challenges incorporating novel tools into LMSs to create a supportive and engaging learning environment. MoodleBot addresses this challenge by offering an interactive platform for both students and teachers. Research Questions: Despite issues like bias, hallucinations, and teachers’ and educators’ resistance to embracing new (AI) technologies, this research investigates two questions: (RQ1) To what extent do students accept MoodleBot as a valuable tool for learning support? (RQ2) How accurately does MoodleBot churn out responses, and how congruent are these with the established course content? Methodology: This study reviews pedagogical literature on AI-driven chatbots and adopts the retrieval-augmented generation (RAG) approach for MoodleBot’s design and data processing. The technology acceptance model (TAM) evaluates user acceptance through constructs like perceived usefulness (PU) and Ease of Use. Forty-six students participated, with 30 completing the TAM questionnaire. Findings: LLM-based chatbots like MoodleBot can significantly improve the teaching and learning process. This study revealed a high accuracy rate (88%) in providing course-related assistance. Positive responses from students attest to the efficacy and applicability of AI-driven educational tools. These findings indicate that educational chatbots are suitable for integration into courses to improve personalized learning and reduce teacher administrative burden, although improvements in automated fact-checking are needed.