Spatiotemporal masked pre-training for advancing crop mapping on satellite image time series with limited labels

Abstract

Accurate crop mapping plays a critical role in optimizing agricultural monitoring and ensuring food security. Although data-driven deep learning methods have demonstrated success in crop mapping with satellite image time series (SITS) data, their promising performances heavily depend on labeled training samples. Nevertheless, the difficulty of annotating crop types often results in labeled data scarcity, leading to a decline in the model’s performance. Self-supervised learning (SSL) is a novel technique for crop mapping with limited labels. However, the existing SSL methods applied to SITS data typically explore masking solely on temporal dimension, which cannot guarantee strong spatial representation and therefore hinders the accurate prediction of complex crop fields. Furthermore, these methods sequentially extract spatial and temporal information without fully integrating information across different dimensions. In this study, we propose a spatiotemporal masking strategy for pre-training a SpatioTemporal Collaborative Learning Network (STCLN) to extract informative spatial and temporal representations from SITS data. Additionally, we design a SpatioTemporal Attention (STA) module in STCLN that integrates representations from spatial and temporal dimensions. The experimental results on two crop type mapping benchmarks encompassing various crop types demonstrate the outperformance of our proposed method. STCLN_wp outperforms the previous state-of-the-art (SOTA) methods with 6.49% higher mIoU on PASTIS dataset and 4.04% higher mIoU on MTLCC dataset. The ablation experiments on pre-training, masking strategies, and the STA module validate the effectiveness of our methodological design. Additionally, experiments conducted under varying sizes of the training set highlight the superior generalization ability of our method for crop type mapping in label-scarce situations. The code of our method is available at https://github.com/XiaoleiQinn/STCLN.