How Do Machine Learning Algorithms Perform in Predicting Hospital Choices?: Evidence from Changing Environments

Abstract

The proliferation of rich consumer-level datasets has led to the rise of the "algorithmic modeling culture" [2] wherein analysts treat the statistical model as a "black box" and predict choices using algorithms trained on existing datasets. In most cases, these evaluations of algorithmic prediction have focused on settings where individuals face the same choices over time. However, evaluating policy questions often involves modeling a substantial shift in the choice environment. For example, a health insurance reform may change the set of insurance products that consumers can buy, or a merger may alter the products available in the marketplace. For such questions, it is less obvious whether machine learning methods can usefully be applied. As Athey [1] remarks: [M]uch less attention has been paid to the limitations of pure prediction methods. When SML [supervised machine learning] applications are used "off the shelf" without understanding the underlying assumptions or ensuring that conditions like stability [of the environment] are met, then the validity and usefulness of the conclusions can be compromised.