MVPA Permutation Schemes: Permutation Testing in the Land of Cross-Validation

Abstract

Permutation tests are widely used for significance testing in classification-based fMRI analyses, but the precise manner of relabeling varies, and is generally non-trivial for MVPA because of the complex data structure. Here, we describe two common means of carrying out permutation tests. In the first, which we call the "dataset-wise" scheme, the examples are relabeled prior to conducting the cross-validation, while in the second, the "fold-wise" scheme, each fold of the cross-validation is relabeled independently. While the dataset-wise scheme maintains more of the true dataset's structure, additional work is needed to determine which method should be preferred in practice, since the two methods often result in different null distributions (and so p-values).