For those making group or individual judgments or decisions, on the job or personally: when Danny Kahneman has something to say, listen and learn.
His latest book, co-authored with Olivier Sibony and Cass Sunstein, is “Noise: A Flaw in Human Judgment” (Little Brown Spark, May 2021) –- deservedly long-listed by the Financial Times for business book of the year.
“Noise” extends a history of scholarship that goes back to Kahneman’s ground-breaking essay with Amos Tversky, his long-time and much mourned colleague, “Judgment Under Uncertainty: Heuristics and Biases” (Science, vol. 185, 1974), and was surveyed comprehensively in his encyclopedic work, “Thinking, Fast and Slow” (Farrar, Straus and Giroux 2011).
Standing on the broad shoulders of Kahneman and Tversky, and the two generations of scholars and practitioners who have refined and applied their insights, we look over a landscape of now-familiar concepts -- representativeness, familiarity and anchoring; sunk costs and loss aversion; misaligned incentives and the dangers of not knowing what you don’t know.
Studies of judgment errors have focused over a half-century on bias, with multitudes of strategies for its identification and reduction. “Noise” now offers an awareness of something distinctly different –- the all-too-frequent, under-recognized and undesirable variability of human judgments, in decisions and outcomes that are sub-optimal and costly at best and deeply unfair or unjust at worst.
A selection among the many colorful examples:
- The wide and perplexing range of prison sentences imposed for the same crimes by different judges, or even by the same judge.
- The amount of disagreement among diagnostic physicians, assessing the same case.
- The high-low spread of insurance premiums quoted by different underwriters for identical risks.
- The divergence of “expert” predictions on future events, from movie sales to interest rates to election outcomes.
The authors bring several surprises – one, that although decision-makers have been frustrated by the degree of noisy variability, they have not (until now) treated it as a subject for identification, analysis and pursuit of responses.
Another, the commonality of noise-inducing conditions across multiple environments:
- Single decisions are as much infected by noise as those that are repetitive or frequent.
- Individuals and groups or teams are similarly susceptible.
- Noise is no less present in predictive decisions whose future outcomes are unknowable, as in choices where simple research or fact-checking would have been relevant and valuable.
The authors readily recognize that variability in judgments is often desirable. Diversity of creative approaches and contrarian opinions are important across fields from film making to market making. But while diversity of views may benefit entrepreneurship and risk taking, it is detrimental elsewhere. If two doctors disagree on a diagnosis, at least one of them is wrong; errors in underwriting do not cancel or average out, but are cumulative; it does not inspire confidence that a job offer or a performance evaluation can be skewed by the time of day it is decided.
Space allows only a sampling of the authors’ analytic taxonomy:
- System noise is broken down between level noise – the variability of judgments due to process ambiguities – and pattern noise – differences in results based on the personal predilections of the decision-makers.
- A noise audit is proposed as the starting point for an organization’s systemic commitment to a noise-reduction effort.
- Decision hygiene encompasses noise reduction by analogy to hand washing in a hospital, reducing the presence of errors, even –- like germs -- without their precise identification.
“Noise” commends tailored and specific strategies to address and reduce unwanted variability, recognizing that the balance between the predictability of strict application and enforcement of rigid rules, and the elasticity of general standards at the discretion of an enforcement authority, is fraught with complexity. It is also at pains to recognize the necessity for trade-offs -- two in particular:
First, costs and benefits require evaluation. As an example, difficult as it is to predict the eventual success of newly hired employees, HR decisions can be improved, at least up to a point. Results improve with inputs from multiple interviewers, provided independence is preserved and the process is not subverted by committee influence or the urgings of a vocal advocate, or where sequential evaluations are passed along, so that an early rating takes on undue influence.
Similarly, noise in the grading of student work owes to the vagueness of evaluation criteria and the fatigue factor or even work hours of the grader. Research shows improvement where essays are ranked best-to-weakest –- a technique more dependable than the assignment of individual numeric scores. Having more than one grader reduces variability, although the time and cost are more easily justified for university admissions or graduate theses than for fifth graders.
Second, reduction in unwanted variability can implicate fairness and equity. Criminal sentencing, long and widely criticized as so unpredictable as to be a “lottery,” depends not only on the judge assigned, but also on such seemingly irrational factors as the time of day or day of the week of a sentencing hearing.
Which said, judges themselves and other critics have reacted negatively to rigid sentencing guidelines, with “three strikes” thresholds and mandatory prison terms, as removing discretion to consider extenuating circumstances or the prospects for rehabilitation.
That challenge was on display at my boarding school, a very long time ago. The slender Students Handbook was simplicity itself. Strict prohibitions were few: curfews were explicit, attendance and punctuality were expected and enforced, alcohol and drugs were banned. Beyond that, judgment was expected. Alumni of those years recall and recite the central but unwritten principle, “There are no rules here – until you break them” – at which point, official wrath was swift, unambiguous, and unappealable.
Modern examples are available. Readers who share my long-running concern for the model by which assurance is provided on the financial statements of the world’s large public companies will know first-hand the impossibility of divining predictability, comparability or certainty from the pronouncements of purported authorities in such areas as accounting and reporting standards and auditor performance quality.
With its attention to the necessity for balancing and trade-offs, “Noise” has a somewhat provisional feel, especially compared with the notorious rigor that characterized the scholarship of team Kahneman/Tversky, who were unwilling to publish until they had tested themselves and their ideas in protracted scrutiny and debate. Instead, freely admitting the uncertainties and acknowledging the difficulty of generalizing across different domains, the authors both warn readers to proceed with caution and skepticism, and make explicit calls for further scholarship.
A message for my graduate students in Risk Management and Decision-Making who are staying in post-classroom contact: add “Noise” to your reading list, along with my other extra-curricular recommendations.
It’s not a summer beach book. But it will be rewarding, as you extend your studies and as your careers take shape.
(To other readers, if interested in the scope and themes of this course –- for questions, or a syllabus or reading list, drop me an email.)
The affectionate dual biography of Kahneman and Tversky by Michael Lewis, “The Undoing Project” (W.W.Norton 2016), includes the former’s simple test and measure of intelligence: the smarter you were, the faster you realized that Tversky was smarter than you. Modified here, for the sympathetic who have already over-come the natural resistance that inhibits best quality decision-making:
The concepts laid out in “Noise” are complex and provocative. The sooner those responsible for consequential decision-making become engaged, the sooner may come valuable changes in areas previously deemed intractable.
Thanks for joining this dialog. Please share with friends and colleagues. Comments are invited and welcome, and subscription sign-up is easy and free, both at the Main page.