Using Deep Learning to Track Representational Flexibility Development of Children with Autism in a Virtual World

Abstract

Representational Flexibility (RF) is an individual cognitive ability to select, coordinate and create alternative representations during information processing. Students with autism spectrum disorder (ASD) typically lack representational flexibility and therefore are underrepresented in the field of stem education. Research has used virtual reality-based (VR) 3D simulations to promote RF development in ASD adolescent learners. However, to promote RF development through VR, the need to be able to track RF throughout the sessions is critical. In this paper we describe the development of neural network classifiers, designed to track RF subsets throughout the session. We trained these classifies using different modes of data collected from 178 session recordings with eight ASD participants. Our empirical results show the promise and potential of using deep learning to provide real-time tracking of the cognitive-affective states of students with ASD. To the best of our knowledge, this is the first research effort to combine VR-training, educational data mining and deep learning to provide a nurturing and adaptive learning environment for students with ASD.