Foreign objects in coal mining and washing operations pose significant challenges, including equipment wear, production inefficiencies, and safety hazards. Current sorting methods, predominantly manual or based on Horizontal Bounding Box detection, struggle to meet the requirements of dynamic environments due to their inability to accurately predict target orientation and suppress background interference. This study introduces YOLOv5-SROD, a rotational object detection algorithm tailored for foreign object detection on vibrating screens. The model introduces rotating bounding boxes with a Circular Smooth Label strategy, ensuring stable and accurate angle predictions while addressing challenges such as angle jumping. Additionally, the Squeeze-and-Excitation attention mechanism enhances feature extraction in complex scenarios by suppressing noise from reflective water spray and high-glare conditions. Experimental results reveal that YOLOv5-SROD achieves a [email protected] of 84.5%, processes at 30.4 FPS, and features a lightweight design with 21.68 million parameters, outperforming both HBB methods and state-of-the-art rotational detection models. These results highlight YOLOv5-SROD’s capability to deliver real-time, accurate detection in challenging industrial environments, offering a scalable and practical solution for foreign object detection in coal preparation processes.