Silicone rubber dynamic seals used in abrasive contaminated environments often fail due to fretting wear of silicone rubber. Analysis of the fretting wear mechanism for silicone rubber containing hard abrasive particles on the contact interface can improve the safety in service and reliability design of silicone rubber seals. The fretting wear behavior of silicone rubber is experimentally investigated under quartz sand abrasives. Five different abrasive concentrations and four different abrasive sizes (26–40 mesh, 40–60 mesh, 60–80 mesh, and 110–160 mesh) of quartz sand abrasives are considered. The sphere-on-flat contact between 440C steel ball and silicone rubber coating is used as the contact model in fretting wear experiments. Fretting wear experiments are conducted by the reciprocating friction wear tester under different experiment conditions, including normal force, displacement amplitude, and frequency. Three-dimensional white-light interference profilometer and scanning electron microscope are utilized to measure wear volume and fretting wear morphology. The fretting state is analyzed based on the friction curves regarding relative tangential displacement and tangential force. The influences of the abrasive concentration, displacement amplitude, abrasive size, frequency, and normal force on the fretting state, coefficient of friction, wear volume, damage characteristic, and fretting wear performance are studied in detail. The results show that the introduction of a small amount of quartz sand abrasives on the contact surface can improve wear resistance of silicone rubber. But excessive quartz sand abrasives can lead to poor wear resistance of silicone rubber when the displacement amplitude or normal force is large.