Theory and the Replication Problem

Many refinements of statistical design have been offered to solve the replication problem identified by the Open Science Collaboration and Camerer and colleagues. There are, however, two distinct kinds of experimentation: Fisher design and theory designed. Therefore, there are two kinds of replication. Only for the Fisher design does replication reproduce conditions of prior experiments in order to compare new with prior results, and only there has a replication problem been demonstrated. In contrast, replications for theory-designed experiments test experimental results against theoretical predictions, and only for theory-designed experiments can replication be extended broadly across the scope of a theory. The authors analyze the logic of the two types of experiments as well as hybrids that mix qualities of both.