How does training improve individual forecasts? Modeling differences in compensatory and non-compensatory biases in geopolitical forecasts

Abstract

Biases in human forecasters lead to poor calibration. We assess how formal training affects two types of bias in probabilistic forecasts of binary outcomes. Compensatory bias occurs when underestimation in one range of probabilities (e.g., less than 50%) is accompanied by overestimation in the opposite range. Non-compensatory bias occurs when the direction of misestimation is consistent throughout the entire range of probabilities. We present a new approach to modeling probabilistic forecasts to determine the extent and direction of compensatory and non-compensatory biases. Using data from the Good Judgment Project, we model the effects of training (randomly assigned) on the calibration of 39,481 initial forecasts from 851 forecasters across two years of the contest. The forecasts exhibit significant indications of both compensatory and non-compensatory biases across all forecasters. Training significantly reduces the compensatory bias in both years. It reduces the non-compensatory bias only in the second year of the contest.