An urgent call for robust statistical methods in reliable feature importance analysis across machine learning

Abstract

Accurate analytical outcomes in machine learning are contingent on error-free calculations and a solid understanding of foundational principles. A notable challenge arises from the lack of ground truth values for validation, complicating the assessment of feature importance, especially when employing linear models with parametric assumptions. This paper critiques the use of Pearson correlation and feature importances derived from Gradient Boosting Regressor (GBR), emphasizing their limitations in analyzing nonlinear and nonparametric data. We propose robust statistical methods, such as Spearman’s correlation and Kendall’s tau, as alternatives for capturing complex relationships while providing essential directional information. Additionally, attention to Variance Inflation Factor (VIF) is crucial for mitigating feature inflation. By addressing these concerns, researchers can achieve more reliable analyses and deeper insight into variable relationships.