Adjusting Distributed Cameras for Robust Moving Object Pose Estimation

Abstract

Robust moving object pose estimation is crucial in fine manipulation tasks, such as surgical instrument tracking. This paper presents a distributed-camera system with robotic adjustments to maintain consistent tracking of moving objects, thus avoiding tracking failures. An integrated framework for camera adjustment and pose estimation is developed for this distributed-camera system. In each detection cycle, the camera exhibiting the largest deviation with the object is adjusted by a visual servoing technique. After adjustment, the camera extrinsics are re-calibrated in the following detection cycles. For the unadjusted cameras, an online extrinsic optimization method based on multi-frame detection results is proposed to refine the camera extrinsics. Based on the refined camera extrinsics and detection results from multiple cameras, the pose of moving objects relative to the principal camera can be robustly estimated. We test the performance of this system in both simulation environments and real-world scenarios. The results indicate that our system achieves higher pose estimation accuracy and exhibits strong resistance to limited field-of-view (FoV) compared to conventional equivalent fixed multi-camera systems. Note to Practitioners—The motivation of this work is to tackle the challenge of FoV limitation during moving object pose estimation. Most optical tracking systems address this issue through the fusion of multiple sensors. However, their optical tracking devices are typically fixedly installed, which can lead to failures in pose estimation when objects move out of the device’s FoV. To expand the tracking scopes while simultaneously ensuring the pose estimation accuracy, we proposed to apply narrow-FoV cameras of which the locations can be adjusted by gimbals to track moving objects, keeping them at the center of the FoV. In this way, these cameras can always detect points with a wider distribution at the center of the FoV, which is conducive to improving detection accuracy. Our proposed system achieves a significant resistance towards FoV limitation during the pose estimation process of objects in large-scale motion.