Handwritten character classification from EEG through continuous kinematic decoding

Abstract

The classification of handwritten letters from invasive neural signals has lately been subject of research to restore communication abilities in people with limited movement capacities. This study explores the classification of ten letters (a,d,e,f,j,n,o,s,t,v) from non-invasive neural signals of 20 participants, offering new insights into the neural correlates of handwriting. Letters were classified with two methods: the direct classification from low-frequency and broadband electroencephalogram (EEG) and a two-step approach comprising the continuous decoding of hand kinematics and the application of those in subsequent classification. The two-step approach poses a novel application of continuous movement decoding for the classification of letters from EEG. When using low-frequency EEG, results show moderate accuracies of 23.1% for ten letters and 39.0% for a subset of five letters with highest discriminability of the trajectories. The two-step approach yielded significantly higher performances of 26.2% for ten letters and 46.7% for the subset of five letters. Hand kinematics could be reconstructed with a correlation of 0.10 to 0.57 (average chance level: 0.04) between the decoded and original kinematic. The study shows the general feasibility of extracting handwritten letters from non-invasively recorded neural signals and indicates that the proposed two-step approach can improve performances. As an exploratory investigation of the neural mechanisms of handwriting in EEG, we found significant influence of the written letter on the low-frequency components of neural signals. Differences between letters occurred mostly in central and occipital channels. Further, our results suggest movement speed as the most informative kinematic for the decoding of short hand movements.