Semi-automatic 3-D pose estimation of laparoscopic tools to generate 3-D labeled database by developing a graphical user interface

Abstract

With medical science advancement in today’s modern world, minimally invasive surgery (MIS) has many advantages over open surgery. Despite these advantages, this method also has problems that can be resolved with 3-D surgical tools pose estimation by using a graphical user interface (GUI) to generate a 3-D labeled database of minimally invasive surgery. Since surgery is a continuous act, we have to consider the tools’ position in each frame of the surgery video to estimate the 3-D pose of the tools during the surgery correctly. Previous studies to find the tool’s position have been based on recognizing the tool in the image and then estimating its position by different methods in two or three dimensions. Since each of these methods had errors, we looked to reduce the errors and find the 3D position corresponding to each tool semi-automatically and more accurately. To this end, we develop and design a graphical user interface that displays the surgical environment in three dimensions. We also design the 3-D models of the laparoscopic tools so that by registering them on the 2-D images of the tools during surgery, the instrument’s position could be identified more accurately. To register 3-D models on 2-D images of them, we use pre-prepared data that identifies each type of tool and each part of it with a distinct color. Finally, we can find the tool’s location in the image and its placement using the spatial averaging of the color of each tool and its components. Next, we intend to match the models we have simulated in a different environment to the 2-D images of the instruments by automatically recognizing the tool’s type using the user’s knowledge. At this point, according to the unique color of each tool and its components, as well as the coordinates of placement of these colors, we determine the location of the tool and the axis of the tool to determine the angle of the tool in two dimensions. Finally, we position the tool model on the image manually and see the percentage of similarity using Sum of Absolute Differences method. Besides, this adjustment becomes more accurate by automatic checking for the best percentage of similarity while the tool rotates around the third axis. In conclusion, this method can generate a labeled database, which would help us use more accurate methods (such as using neural networks) to find the 3-D pose of surgical tools.