Task recognition integrating worker actions and machine operations: A video-based sensing approach without physical sensors

Abstract

Automating work process analysis is crucial in manufacturing to improve efficiency and productivity. However, traditional deep learning methods often fail to capture subtle temporal changes in machine operations, such as varying speeds. We propose a cost-effective approach called pseudo-sensing, which simulates sensor data by measuring machine speeds directly from video using wavelet transformation, a mathematical tool for time-frequency analysis. This approach eliminates the need for physical sensors.

We evaluated pseudo-sensing by integrating it into two task classification models. The first is a convolutional neural network-long short-term memory (CNN-LSTM) model, which extracts spatial features via a CNN and learns temporal patterns using an LSTM. The second is a three-dimensional residual network (3D ResNet, R3D), designed to process spatiotemporal data simultaneously. With pseudo-sensing, the CNN-LSTM’s micro-F1 score—an accuracy metric averaging precision and recall across all classes—improved from 0.712 to 0.736 (+2.4 points), while R3D’s score rose from 0.675 to 0.701 (+2.7 points).

To assess general applicability, we tested pseudo-sensing on another dataset featuring diverse machine motions: unidirectional movements (e.g., conveyor belts), oscillatory movements (e.g., pendulum-like motions), rotational movements (e.g., rotary presses), and intermittent movements (e.g., blinking or toggling mechanisms). The method achieved an 83% success rate in identifying machine dynamics.

By leveraging deep learning, this method integrates video-based machine operation sensing with task recognition, considering both human actions and machine states. Eliminating additional sensors while enhancing accuracy and efficiency, pseudo-sensing offers broad potential for advancing manufacturing process analysis.