SegTrackDetect: A window-based framework for tiny object detection via semantic segmentation and tracking

Abstract

This work introduces SegTrackDetect, an open-source, window-based framework for small and tiny object detection. Detecting tiny objects in high-resolution images is essential for real-time applications such as autonomous navigation and surveillance but is challenging due to the computational complexity of processing large images while maintaining the speed needed for timely decision-making. The proposed framework addresses this challenge by performing inference in selected regions only, significantly reducing the computational burden compared to standard sliding window methods. Thanks to full-resolution inference within these selected regions, lightweight detectors can be employed, further accelerating the process. The framework selects detection sub-windows based on Regions of Interest (ROIs) generated by the ROI Estimation and ROI Prediction modules. The ROI Estimation Module creates binary masks of ROIs from input images, while the ROI Prediction Module uses an object tracker to predict object locations in the current frame based on previous detections. Detections from multiple sub-windows are aggregated and filtered to eliminate redundancies, ensuring high-quality results. SegTrackDetect is optimized with inference speed in mind, offering a highly efficient pipeline while providing users with the flexibility to customize models. It supports a wide range of industrial and research applications by allowing users to adjust model parameters and incorporate new models with custom pre- and post-processing functions. It is compatible with both image and video data, automatically determining the data type from the dataset structure. SegTrackDetect is especially well-suited for tasks such as drone or satellite-based tiny object detection. The code is available at https://github.com/deepdrivepl/SegTrackDetect.