Learning feature transforms is an easier problem than feature selection

We argue that optimal feature selection is intrinsically a harder problem than learning discriminative feature transforms, provided a suitable criterion for the latter. We discuss mutual information between class labels and transformed features as such a criterion. Instead of Shannon's definition we use measures based on Renyi entropy, which lends itself into an efficient implementation and an interpretation of "information forces" induced by samples of data that drive the transform.