On Relevant Features Selection Based on Information Theory

It is shown that widely used suboptimal algorithms of feature selection based on information theory concepts do not necessarily identify a collection of features (relevant in a sense) affecting the studied random response. This can be considered as a reflection of the epistasis phenomenon known in genetics, when individual features have little effect on increased risk of complex disease, whereas certain combinations of features have significant impact on risk. It is demonstrated that a similar effect is also manifested in inferences employing statistical estimates of mutual information.