Cow depth image restoration method based on RGB guided network with modulation branch in the cowshed environment

Abstract

Depth images were widely applied in smart animal husbandry. The raw depth images collected by the RGB-D cameras generally existed amount of missing depth values due to the light reflected from white pattern of cows and direct sunlight in the cowshed. The incomplete cows in depth images would affect the application of depth images in health monitoring. This study proposed a cow depth image restoration method based on RGB guided network with a modulation branch. Firstly, removing the outliers caused by light from the depth image and determining the depth value missing area of the cow’s body. Second, RGB and depth features were extracted through multiple convolutions and fused in the S-C (Self-attention and Convolution attention) fusion module of encoder. Then, the prediction head generated a coarsely predicted depth image after deconvolution combined with a modulation branch. Finally, the repaired depth image was generated in the SPN (Spatial Propagation Network) refinement module of the decoder. In terms of dataset construction, 7260 depth images were collected in a commercial dairy farm. To make up for lacking ground truth complete depth images corresponded to the raw depth images with missing value, two ways for generating missing depth images were designed. The experimental results shown that the method had improved restoration quality of cow’s incomplete body in depth images. By comparing with other depth restoration works, the proposed method achieved significantly superior performance on RMSE = 36.32 and MAE = 12.77, and the percentage of predicted pixels within the error range at 1.25 reached 0.999. Additionally, a smoother transition between missing and restoration regions was demonstrated in the repaired depth images and point cloud results. And compared with the depth images with missing regions, the Precision, Recall rate and F1-score of the repaired depth images were improved for cow body condition scoring. This study could improve the effectiveness of the collected data and make the depth images more practical for smart animal husbandry.