Benchmark Meta-Dataset of High-Resolution Remote Sensing Imagery for Training Robust Deep Learning Models in Machine-Assisted Visual Analytics

Abstract

Recent years have seen the publication of various high-resolution remote sensing imagery benchmark datasets. These datasets, while diverse in design, have many co-occurring object classes that are of interest for various application domains of Earth observation. In this research, we present our evaluation of a new meta-benchmark dataset combining object classes from the UC Merced, WHU-RS19, PatternNet, and RESISC-45 benchmark datasets. We provide open-source resources to acquire the individual benchmark datasets and then agglomerate them into a new meta-dataset (MDS). Prior research has shown that contemporary deep convolutional neural networks are able to achieve cross-validation accuracies in the range of 95-100% for the 33 identified object classes. Our analysis shows that the overall accuracy for all object classes from these benchmarks is approximately 98.6%. In this work, we investigate the utility of agglomerating the benchmarks into an MDS to train more generalizable, and therefore translatable from lab to real-world, deep machine learning (DML) models. We evaluate numerous state-of-the-art architectures, as well as our data-driven DML model fusion techniques. Finally, we compare MDS performance with that of the benchmark datasets to evaluate the performance versus cost trade-off of using multiple DML in an ensemble system.