A Semantic Hybrid Temporal Approach for Detecting Driver Mental Fatigue

Driver mental fatigue is considered a major factor affecting driver behavior that may result in fatal accidents. Several approaches are addressed in the literature to detect fatigue behavior in a timely manner through either physiological or in-vehicle measurement methods. However, the literature lacks the implementation of hybrid approaches that combine the strength of individual approaches to develop a robust fatigue detection system. In this regard, a hybrid temporal approach is proposed in this paper to detect driver mental fatigue through the combination of driver postural configuration with vehicle longitudinal and lateral behavior on a study sample of 34 diverse participants. A novel fully adaptive symbolic aggregate approximation (faSAX) algorithm is proposed, which adaptively segments and assigns symbols to the segmented time-variant fatigue patterns according to the discrepancy in postural behavior and vehicle parameters. These multivariate symbols are then combined to prepare the bag of words (text format dataset), which is further processed to generate a semantic report of the driver’s status and vehicle situations. The report is then analyzed by a natural language processing scheme working as a sequence-to-label classifier that detects the driver’s mental state and a possible outcome of the vehicle situation. The ground truth of report formation is validated against measurements of mental fatigue through brain signals. The experimental results show that the proposed hybrid system successfully detects time-variant driver mental fatigue and drowsiness states, along with vehicle situations, with an accuracy of 99.6% compared to state-of-the-art systems. The limitations of the current work and directions for future research are also explored.