An Intelligent Microscope Vision System for Hooking Multi-Thread Flexible Microelectrode Based on Few-Shot Segmentation

Abstract

Multi-thread flexible microelectrodes hold the promise of high-quality and long-term neural signal recording. To implant fragile microelectrodes into brain, a microneedle is usually used as a shuttle to increase stiffness. However, hooking microelectrodes with a microneedle is error-prone and time-consuming, due to the difficulty in accurately obtaining micro-object coordinates through low-quality microscope images. To solve this problem, this paper proposes an intelligent microscope vision system based on few-shot segmentation method. Firstly, a stereo vision system is designed to enable cameras to focus on all microelectrode threads simultaneously. Secondly, a few-shot instance segmentation method is proposed to obtain key-point coordinates for vision guidance, which consists of data augmentation, improved Mask-RCNN and anchor-free clustering loss. The data augmentation method synthesizes low-quality images to strengthen model’s generalization ability. To improve the proposal quality, a residual region proposal module is introduced into Mask-RCNN. By clustering instance features in metric space, the anchor-free clustering loss enhances the model capability of predicting hard instances and avoids the intra-class bias of fixed anchor. Experimental result shows that the proposed method achieves 99.72% and 90% correct rate for microelectrode and microneedle segmentation with 5-shot training. Code is available at https://github.com/NamingIsEasy/FSS_implantation. Note to Practitioners—Multi-thread flexible microelectrodes are an emerging brain-computer interface platform. Compared with traditional electrodes, they have the advantages of low elastic modulus, micron-level width and material biocompatibility. However, the low elastic modulus also makes it difficult to be inserted into brain. Thus, researchers proposed “sewing machine” paradigm, in which a microneedle is used to hook microelectrodes and drive them to be inserted into brain. The hooking process is threading the microneedle into engaging holes at the end of microelectrodes. However, undesirable conditions in microscope images, such as out-of-focus blur and occlusion, make it difficult to accurately obtain microneedle and microelectrode coordinates. Towards automated hooking operation, this paper proposed an intelligent microscope vision system based on few-shot segmentation. Firstly, a stereo vision system is designed to focus cameras on all microelectrode threads simultaneously. Secondly, a few-shot segmentation method is proposed to reduce time of image acquisition and annotation process. It consists of data augmentation, improved Mask-RCNN and anchor-free clustering loss. The data augmentation method synthesizes undesirable-condition images to reduce the bias of training dataset. The classical region proposal module in Mask-RCNN is replaced with an improved residual version. To make the model focus more on hard instances during training, an anchor-free clustering loss is introduced. The proposed method achieves 99.72% and 90% correct rate for microelectrode and microneedle segmentation in 5-shot experiments. In future, we will explore automated hooking control and subsequent implantation process based on the proposed vision system.