Investigating Challenges and Opportunities of the Touchless Hand Interaction and Machine Learning Agents to Support Kinesthetic Learning in Augmented Reality

Abstract

Augmented Reality (AR), with its potential to bridge the virtual and real environments, creates new possibilities to develop more engaging and productive learning experiences. Evidence is beginning to emerge that this sophisticated technology offers new ways to improve the learning process for better interaction and engagement with students. Recently, AR has garnered much attention as an interactive technology that facilitates direct interaction with virtual objects in the real world. These virtual objects can be surrogates for real world teaching resources, allowing for virtual labs. Thus AR could allow learning experiences that would not be possible in impoverished educational systems around the world. Interestingly though, concepts such as virtual hand interaction and techniques such as machine learning are still not widely investigated in the domain of AR learning. The need for touchless interaction technologies has exceptionally increased in this post-COVID world. There are also existing pedagogical approaches that have not been explored in great detail in this new medium, such as Kinesthetic learning or ”Learning by Doing”. Using the touchless interaction hand interaction technology and machine learning agents, this research aims to address this gap by exploring these underutilised technologies to demonstrate the efficiency of AR learning. It will explore the different hand tracking APIs to integrate the virtual hand interaction, testing the devices’ compatibility with these APIs and integrating machine learning agents using reinforcement learning to develop an AR learning framework that can provide more productive and interactive learning experiences.