Integrative human and object aware online progress observation for human-centric augmented reality assembly

Augmented reality (AR) can provide step-by-step intuitive guidance for workers on the shop floor, enabling time-saving and error-avoid assembly actions. Nevertheless, existing AR-guided assembly methods have primarily paid attention to information on assembly objects and usually ignore the human factor in the assembly process. Further, there are a series of details regarding the AR system design that are frequently neglected, including systematic usability, human intervention, and AR perspective. To alleviate these limitations, this paper proposes a real-time two-branch approach that integrates human action-based human factor evaluation and object-based assembly progress observation. In the online human factor evaluation, a skeleton-based model is applied to predict the operator’s assembly action, providing a quantitative analysis and optimized indicator for the ongoing AR assembly. In the assembly progress observation, the object-based model is deployed to recognize the assembly part, and the AR assembly status is checked automatically based on the prior sequential assembly knowledge without human intervention. Thus, a holistic human-object integrated framework is established for the human-centric AR assembly process inspection, as well as the quantitative analysis and optimized indicator output from the framework are actively feedback in the first-person AR perspective, where the operators can perceive the assembly stage and whether their working posture is appropriate or not intuitively. Finally, extensive experiments are carried out on the human-object integrated performance in the smart AR assembly, and results illustrate that the proposed method can monitor the online assembly observation from a holistic perspective, alleviate the cognitive load, and achieve superior performance for the AR assembly tasks.