Quantifying the Unquantifiable

Abstract

Every day, countless experts offer innumerable opinions in a dizzying array of forums. Cynics groan that expert communities seem ready at hand for virtually any issue in the political spotlight—communities from which governments or their critics can mobilize platoons of pundits to make prepackaged cases on a moment’s notice. Although there is nothing odd about experts playing prominent roles in debates, it is odd to keep score, to track expert performance against explicit benchmarks of accuracy and rigor. And that is what I have strug­ gled to do in twenty years of research of soliciting and scoring experts’ judgments on a wide range of issues. The key term is “struggled.” For, if it were easy to set standards for judging judgment that would be hon­ ored across the opinion spectrum and not glibly dismissed as another sneaky effort to seize the high ground for a favorite cause, someone would have patented the process long ago.