Method Development Through Landmark Point Extraction for Gesture Classification With Computer Vision and MediaPipe

Abstract

Examining the physical movements of students during their educational quests holds great significance as these nonverbal cues can exert a substantial influence on academic performance, and boost, learning outcomes, Consequently, numerous researchers are engaged in exploring the domain of gesture categorization employing machine learning techniques. Initially, we conducted an observation of students’ movements in a virtual learning environment during face-to-face interactions with their teachers. This procedure yielded a roster of thirteen motion-based behaviors, encompassing actions such as tilting the head towards either direction, lowering and lifting the head, as well as gesturing with the right and left hand towards the head and neck area, and positioning the shoulders in a front and lateral direction. This research offers a technique for establishing a set of criteria for categorizing students’ gesticulations in online learning by utilizing the comprehensive MediaPipe holistic library and OpenCV to detect, pose and extract salient landmarks. This endeavor culminated in the attainment of a percentage-based metric indicative of gesture identification efficacy pertaining to the aforementioned thirteen motion-based activities.