Human–machine cooperative decision-making and planning for automated vehicles using spatial projection of hand gestures

Significant challenges in perception, prediction, and decision-making within self-driving systems remain inadequately addressed. Concurrently, the advancement of autonomous driving technologies reduces driver engagement, inadvertently eroding their proficiency. Integrating human cognitive flexibility and experiential insight with the machine’s precision and reliability offers a promising approach for the transitional phase towards fully automated driving. This study presents a human-machine collaboration approach to enhance the highly automated vehicles’ high-level flexibility and personalization attribute without the need for passengers’ prior driving experience. Firstly, we propose a tactical human–vehicle collaboration framework leveraging the hand-landmark extraction algorithm and augmented visual feedback. The proposed vision-based interface projects the gesture onto the ground and feeds it back to the driver through the augmented reality head-up display (AR-HUD) for intuitive interaction. The projection offers strategic decision-making guidance and planning recommendations for the vehicle. Utilizing these suggestions, the automation algorithm efficiently manages the remaining tasks, including collision avoidance and adherence to traffic regulations. This approach minimizes the driver’s engagement in routine driving tasks and negates the need for driving skills. Incorporating cooperative game theory, the methodology optimally balances personalization with system robustness. Finally, we compare our approach with conventional manual driving schemes that both can assist the self-driving car in avoiding unknown obstacles and reaching the personalized goal. Results demonstrate that the proposed decision-making and planning collaboration scheme significantly reduces human physical burdens without compromising driving performance and driver mental workloads.