Multi-modal machine learning for the early detection of metabolic disorder in dairy cows using a cloud computing framework

Abstract

In precision livestock farming (PLF), wearable sensors, computer vision, and genomic tests generate large amounts of data, which can be challenging to integrate and analyze jointly due to their diverse nature. However, incorporating both genomic and phenotypic data together can be beneficial for developing predictive models in animal biology. The development of automated and modular data pipelines using scalable solutions such as cloud computing can be an effective strategy to integrate and analyze animal-level information in real-time. The objectives of this study were (1) to propose a cloud computing-based framework to automate the processing and integration of phenotypic and genotypic data, and (2) to assess different data fusion strategies (early and late fusion, and cooperative learning) for the early detection of subclinical ketosis (SCK) in dairy cows, integrating wearable sensors, imaging systems, and genotypic data in livestock farms. We developed a modular pipeline for image analysis, which includes body segmentation, frame quality assessment, animal identification, and body condition score (BCS), which were crucial for producing the features used in SCK detection. The body segmentation module achieved a Dice similarity coefficient of 0.990, the frame quality assessment module reached 99.1 % accuracy, the animal identification module attained 93.2 % accuracy, and the BCS module achieved accuracies of 81.1 % and 96.2 % when allowing up to 0.25 and 0.50 prediction error, respectively. For SCK detection, early fusion and cooperative learning achieved the lowest mean absolute errors in predicting plasma beta-hydroxybutyrate as a continuous variable (as low as 0.242). Late fusion, combined with an ordinary least squares regression, achieved the highest F₁ scores for binary SCK prediction (up to 0.750). These results suggest that data fusion techniques can be effectively used to integrate genotypic and phenotypic data from multiple sensors. Additionally, SCK detection can be performed on dairy farms using the proposed cloud computing-based framework, which was implemented with modular, independent services that can be customized and reused for a variety of tasks.