What is predictable? A commentary on Tetlock et al. (2023)

Abstract

Is prediction possible in world politics—and, if so, when? Tetlock et al. (2023) report some of the first systematic evidence on long-range political forecasting. Asked to guess which countries would get nuclear weapons within 25 years and which would undergo border changes due to war or secession, both experts and educated generalists outperformed chance. On nuclear proliferation—but not border changes—the experts beat the generalists, and the difference grew as the time scale increased from 5 to 25 years. What are we to make of this? The authors see messages for both “skeptics,” who consider the political future irreducibly opaque, and “meliorists,” who acknowledge the difficulties but think expertise can still improve predictions. Moreover, they suggest progress could be made through adversarial collaboration between scholars of the two persuasions, which would push both to specify their priors and adopt falsifiable positions.

It's hard not to admire a research paper that has been more than 25 years in the making—and one can only rejoice that picky referees did not insist the experiment be rerun from scratch. The results prompt two broader questions. First, what makes something easier or harder to predict? Second, when does expertise help? At the risk of restating the obvious, let me offer a few thoughts.

For clarity, consider a task like those in the article.¹ Respondents at time t must guess the value of a variable $Y_{i,t+N}\in \{\mathrm{Yes},\mathrm{No}\}$, N years in the future, for I countries indexed by i. The “success rate” is the proportion of countries for which the respondent chooses correctly. A task of this kind, A, is “easier” for a given individual than another task, B, if that individual's success rate on A tends to be higher than his success rate on B.

When will that be the case? The authors give a few examples of easy and difficult tasks. That New Zealand and Norway will not fight a war is “trivially obvious” (p. 1). That anyone could guess who will be US president in 25 years is “far-fetched” (p. 2). They sought challenges for their respondents that fell within the “Goldilocks zone of difficulty” (p. 2), but they do not say what principles or heuristics guided this choice.

Prediction involves the marriage of information to causal models, explicit or intuitive. This suggests a three-way division of determinants. Difficulty of prediction should depend on: (1) the nature of the underlying causal process, (2) the quality of available models, and (3) the supply of available information.

Getting a high success rate will be easier when the causal process is regular. Most simply, that requires that $Y_{i,t+N}$ have a well-defined mean or linear trend, to which it regresses over time (the process is stationary, in the sense that: $Y_{i,t}=\alpha +\gamma Y_{i,t-1}+\beta X_{i,t}+{\epsilon }_{i,t}$ where $0\le \gamma <1;E\left({\epsilon }_{i,t}\right)=0;Var\left({\epsilon }_{i,t}\right)={\sigma }^2$ and $X_{i,t}$ is a vector of exogenous variables that may include a time trend). Extrapolation from past to future is then possible. With variables that follow a random walk, the value of past information depreciates rapidly. Prediction is also easier if the respondent has a well-tested causal model (in the authors' phrase, “relevant and durable principles of causality… to guide forecasters” (p. 18)). The more tests the model has undergone, the more confidence one can generally have in estimated parameters. Finally, applying a valid model requires information about the values of causal variables, $X_{i,t}$.

These points also suggest when expertise will make a difference. Experts are those who have studied similar cases to develop causal models. When a causal process is highly irregular, study will not help, so regularity is a condition not just for predictability but also for expertise to matter. However, if processes are too simple, even laymen may decode them, so a certain degree of complexity should boost the expertise premium. Models will be more reliable when there has been plenty of opportunity to test them. So another condition is a rich history of similar cases. Finally, expertise should also increase familiarity with relevant information, so experts will have an advantage when information is accessible, but not too easily available to laypeople.

These expectations about when expert analysis will improve prediction overlap with conditions under which subconscious intuition should be most trustworthy. “When do judgments reflect true expertise?” Kahneman (2011) asks. Answer: When the environment is regular, and when the experts have had opportunity “to learn these regularities through prolonged practice.” His example of reliable intuitive judgment is chess; by contrast, “stock picking and long-term political forecasting” are “zero-validity environments.”² Psychology also suggests another type of expertise that may improve forecasting—familiarity with common biases and practice in “taming intuitive predictions.”

Does time frame, N, affect predictability? That we know more about tomorrow than about the distant future is a cliché. Often that's right. Whether a woman will win the 2024 US presidential election is easier to guess than the gender of the 2044 winner. But short-run is not always easier. Whether Joe Biden will be president in 2025 is harder to predict than whether he will be in 2030. If change tends to occur in one direction, the odds of transition may cumulate, eventually reducing uncertainty about the outcome (“in the long run, we are all dead”). Measurable structural factors may over time outweigh unmeasurable contingencies. Estimating the impact of economic development on democratization is easier in 20-year than in 1-year periods (Treisman, 2015). In autocracies with income close to $5000, the best prediction of regime type in year t + 30—“autocracy”—would be right 57% of the time. The best prediction in year t + 60—“democracy”—would have an 81% success rate.³

What about complexity? While this tends to make things harder and increase the expert advantage, the outcomes of complex macro-systems are sometimes easier to predict than microchoices. The level of demand for cars in the US next year may be easier to forecast than which individuals will buy one. Sometimes emergent properties of complex systems are more regular than the individual actions that comprise them.

Some think human actions are less predictable than physical phenomena because of subjectivity. This, too, is not always true. Some human behavior is highly structured, while some physical phenomena are extremely irregular. Among the games humans play, some have single equilibria, making outcomes easy to forecast. Others—often involving asymmetric information and beliefs about the beliefs of others—have multiple equilibria even for the same observable parameters. When many sets of mutually consistent beliefs are possible, it is hard to know which will be “selected.” Expertise might help—for instance, to identify relevant “focal points”—but often it will not. These considerations are summarized in Table 1.

Might these ideas explain the difference in results for nuclear proliferation and border change? Although both causal processes are complex, that for nuclear proliferation involves fewer key players (state governments), and their identities are known, as opposed to secessionist groups that might emerge in future. In both cases, experts will be more familiar with key information than generalists. But whereas nuclear experts tend to know something about all potential nuclear powers, scholars who publish on secession often specialize in limited geographical areas. Few have expertise on both the Kuril Islands and the Ethiopia–Somalia border. Both questions may turn on beliefs about beliefs. But secession—like most processes involving mass mobilization—is particularly prone to multiple equilibria. Few want to join a movement that is too small to be effective, even if many would join one large enough to succeed. Models of such processes go by names like “tipping” or “prairie fires” that evoke the speed with which one equilibrium can replace another (Kuran, 1989; Schelling, 1978).

All these factors suggest why border changes may be harder to predict than nuclear proliferation and less subject to expertise. On the other hand, there are far fewer past instances of the latter from which to learn. Nine states have acquired nuclear weapons, while at least 817 border changes occurred between 1816 and 1996 (Tir et al., 1998). Here, the greater number of cases seems to have been more than offset by the first three considerations. Although prediction in politics is bound to remain difficult, perhaps thinking along these lines may improve our predictions about what can and cannot be predicted.