Computationally Efficient Feature Significance and Importance for Predictive Models

We develop a simple and computationally efficient significance test for the features of a predictive model. Our forward-selection approach applies to any model specification, learning task and variable type. The test is non-asymptotic, straightforward to implement, and does not require model refitting. It identifies the statistically significant features as well as feature interactions of any order in a hierarchical manner, and generates a model-free notion of feature importance. This testing procedure can be used for model and variable selection. Experimental and empirical results illustrate its performance.