Calibrating Legal Judgments

In ordinary life, people who assess other people’s assessments typically take into account the other judgments of those they are assessing in order to calibrate the judgment they are now assessing. The restaurant and hotel rating website TripAdvisor is exemplary, because it facilitates calibration by providing access to a rater’s previous ratings. This makes it possible to see whether a particular rating is coming from a rater who is enthusiastic about every place she patronizes, or from someone who is incessantly hard to please. And even when less systematized, as with the assessment of a letter of recommendation or a college transcript, calibration by recourse to the decisional history of those whose judgments we are assessing is a ubiquitous feature of ordinary life. Yet despite the ubiquity and utility of such calibration, the legal system seems perversely to reject it. Appellate courts do not openly adjust their standard of review based on the previous judgments of the judge whose decision they are reviewing, nor do judges who review legislative or administrative decisions, magistrates who evaluate search warrant representations, and even jurors who assess witness perception. In most legal domains, calibration by reference to the other decisions of the judgment being reviewed is invisible, either because it does not exist or because what reviewing bodies know informally is not something they are willing to admit to using. Appellate courts do not, at least openly, look more carefully at the decisions of a trial judge whose decisions are often reversed, and administrative law judges do not acknowledge examining the decisions of some administrators more closely because of what they know about the decisional history of that administrator. However common it is for ordinary people to attempt to calibrate the decisions of those on whom they rely, the law generally resists such calibration, implicitly prohibiting access to a reviewee’s decisional history and discouraging publicly acknowledging that a decisional history has played a role in a reviewer’s decision. Assisted by insights from cognitive psychology and philosophy, this Article examines law’s seeming aversion to calibration, and to explore what this aversion says about the nature of law and legal decision-making.