Deep learning for surgical workflow analysis: a survey of progresses, limitations, and trends

Automatic surgical workflow analysis, which aims to recognize the ongoing surgical events in videos, is fundamental for developing context-aware computer-assisted systems. This paper reviews representative surgical workflow recognition algorithms based on deep learning, outlining their merits, limitations, and future research directions. The literature survey was performed on three large bibliographic databases, covering 67 lary sources, which were comparatively analyzed in terms of spatial feature modeling, spatio-temporal feature modeling, input pre-processing, regularization and post-processing algorithms, as well as learning strategies. Then, common public datasets and evaluation metrics for surgical workflow recognition are also described in detail. Finally, we discuss all literature from different perspectives, and point out the challenges, possible solutions and future trends. The need for more diverse and larger datasets, the potential of unsupervised and semi-supervised learning approaches, comprehensive and equitable metrics, establishing complete regulatory and data standards, and interoperability will be key challenges in translating models to clinical operating rooms. And we propose that surgical activity anticipation and employing large language model as training assistant are interesting research directions in surgical workflow analysis.