Early fault diagnosis is crucial not only for ensuring the safety and efficiency of power systems but also for averting catastrophic failures and substantial economic losses. Although previous studies have made promising strides, developing an interpretable and dependable diagnostic strategy remains challenging. Over the last four decades, a variety of methods have been proposed to tackle this problem. A wide range of these studies have been undertaken recently on statistical techniques for fault detection and classification, but so far, no definitive method has been identified as the best. Therefore, our principal challenge is to review and classify the types of statistical methods used systematically and to choose the appropriate method in diagnosing faults of traditional and intelligent power systems to strengthen the validity of the research results. To bridge this 'gap', this research provides a systematic review that includes the following: (i) providing an overview of the cause and effect of faults in significant equipment of power systems; (ii) collecting studies pertinent to statistical methods in identifying faults; (iii) selecting fundamental studies to compile a collection of related literature; (iv) organizing the applied statistical tests and techniques for identifying faults according to their approach and framework; (v) a comparative evaluation of the classified techniques; (vi) discussion on how to choose the proper statistical techniques, as well as the consequences of choosing a wrong technique. The findings serve as a guide for engineers, scientists, and researchers, providing insights into the opportunities and challenges for future advancements in the field.