Probability and confidence: How to improve communication of uncertainty about uncertainty in intelligence analysis

Abstract

Intelligence agencies communicate uncertainty to decision-makers through verbal probability phrases that correspond to numerical ranges (i.e., probability lexicons) and ordinal levels of confidence. However, decision-makers may misinterpret the relationship between these concepts and form inappropriate interpretations of intelligence analysts' uncertainty. In two experiments, four ways of conveying second-order probability to decision-makers were compared: (a) probability and confidence phrases written in the text of a report, (b) the addition of a probability lexicon, (c) the addition of a probability lexicon that varied numerical ranges according to the level of confidence (i.e., revised lexicon), and (d) a probability phrase written in text followed by a numerical range that varied according to the level of confidence. The revised lexicon was expected to improve interpretations of second-order probability. The 275 participants in Experiment 1 and 796 participants in Experiment 2 provided numerical estimates corresponding to analytic judgments provided in descriptions about three overseas military operations and also indicated their support for approving or delaying the operations. The results demonstrated that providing the numerical range in the text of the report or providing a probability lexicon, improved interpretations of probability phrases above the verbal phrase-only condition, but not interpretations of confidence. Participants were unable to correctly interpret confidence with respect to the precision of their estimate intervals and their decisions about the operations. However, in Experiments 2 and 3 the effects on these variables of providing decision-makers with information about the source of the analyst's uncertainty were examined. In Experiment 3 (n = 510), providing this information improved correspondence between confidence level and approval of the operation. Recommendations are provided regarding additional methods of improving decision-makers' interpretation of second-order probability conveyed in intelligence reporting.