Book finds expert judgments differ far more than leaders expect

By Books (/journals/books)

In 60 seconds Unwanted scatter among judgments that should match is a major error source organizations almost never measure. At one insurer, 828 underwriters pricing identical cases produced a median gap of 55 percent where leaders expected about 10. Most scatter is pattern noise—idiosyncratic weighting of case features—not a few globally harsh or lenient people. The same expert often shifts across days; group discussion typically amplifies scatter through cascades and polarization rather than canceling it. Decision hygiene—independent sub-scores, private aggregation, and sometimes simple rules—shrinks noise while keeping defined expertise. Consistency still leaves bias intact and can cost mercy, adaptability, and the legitimacy of visibly human calls.

The big idea

Kahneman, Sibony, and Sunstein argue that unwanted scatter among professional judgments that should agree is a source of error at least as costly as bias, yet almost invisible because organizations rarely compare verdicts side by side. Measurable noise audits and structured decision hygiene, they contend, can shrink that lottery without waiting for better people or a known right answer.

The book in brief

At a large insurance company, executives guessed two experienced underwriters would differ by about 10 percent on the same case. A noise audit handing identical files to 828 underwriters found a median gap of 55 percent; claims adjusters showed similarly wide spreads on valuations.

That shock frames the book’s case. Judgment under uncertainty—a sentence, a diagnosis, a premium, a hire—can err two ways. Bias is systematic tilt in one direction. Noise is scatter among people who should agree, or within one person across occasions. Organizations chase bias because it has a story; noise has no direction to blame, so it stays underestimated until someone forces comparison.

The authors decompose the scatter. Level noise is global harshness or leniency. Pattern noise—usually larger—is each judge weighting the same features differently. Occasion noise is mood, fatigue, and information order shifting a single skilled person from day to day. Putting people in a room does not fix this: first speakers trigger cascades, and groups often polarize toward more extreme shared verdicts instead of true averages.

The scatter hides behind the illusion of agreement. Each professional experiences a conclusion as simply reading the case, assumes a peer would match, and almost never sees a rival verdict on the identical file. Ordinary workflow assigns one person per case, so disagreement never surfaces.

Remedies travel under decision hygiene: break complex calls into independent sub-assessments scored before any overall verdict; sequence information so global impressions come last; aggregate private estimates statistically. In some prediction domains, even crude rules beat unaided experts because a formula has no bad day—though people distrust algorithms and punish their errors more harshly than identical human ones.

The closing tension is deliberate. Consistency is not fairness. Killing noise can leave bias untouched, flatten local knowledge, demoralize experts, and strip mercy or adaptability some domains value. The authors say weigh those costs case type by case type rather than let scatter accumulate unseen.

Context

Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein published the book in 2021 to treat inconsistency in professional judgment as the quieter companion problem to cognitive bias. Drawing on sentencing, insurance, hiring, and forecasting evidence, it pushes leaders to treat consistency itself as a measurable management problem.

The key ideas

1

Bias versus noise

Organizations that correct only bias ignore unwanted scatter among judgments that should be identical—a second error source that leaves no direction to blame.

Bias Tight cluster off the center—systematic tilt in one direction

Noise Wide scatter around the target—no single wrong direction

Judgment, for the authors, is any call under uncertainty where reasonable people could disagree: a sentence, a diagnosis, a hiring choice, an insurance premium.

Bias is systematic tilt—the average of many judgments misses in one direction, like a rifle sight set too far left. Noise is variability among judgments that ought to match. Four shooter groups at one bullseye split the idea: tight and centered is accurate; tight but off-center is biased; wide but centered is noisy; wide and off-center is both.

Most organizational talk targets bias because it has a story—a stereotype, an incentive, a blind spot. Noise has none. Two competent, well-intentioned professionals reading the identical file simply land apart, and the same person may reverse an earlier verdict later.

Because scatter points nowhere obvious, leaders underestimate how much they carry until someone deliberately measures side-by-side disagreement.

In practice: Four groups aim at one bullseye; the wide, centered spray is pure noise, and that is the pattern professional systems quietly produce when no one compares files.

2

What a noise audit shows

Giving identical real cases to many professionals independently reveals disagreement no one inside believes exists until the spread appears.

55% median gap between two underwriters on the same case

The book’s central evidence comes from a large insurer where Kahneman advised. Executives thought two experienced underwriters pricing the same case would differ by a tolerable margin.

The audit handed identical files to 828 underwriters without saying consistency was the point. The median difference between randomly paired quotes hit 55 percent—five times what those leaders had guessed. Claims adjusters showed large gaps on valuations too.

A noise audit needs no ground truth. Strip identifying cues, give the same case to many judges alone, and measure disagreement rather than the average. The design generalizes to forecasters on one economic figure, radiologists on one scan, judges on one hypothetical defendant, and patent examiners on one application.

The lesson is systemic, not personal incompetence: without a consistency check, the premium a customer pays or the sentence a defendant receives depends heavily on which employee drew the file.

In practice: 828 underwriters, same stripped cases, independent quotes: median pairwise gap of 55 percent against leadership’s rough 10 percent expectation.

3

Level versus pattern noise

Most disagreement comes from idiosyncratic weighting of case features, not from a few people being globally harsher—so recalibrating outliers leaves the bulk untouched.

Level noise Some judges harsher or more lenient across the board

Pattern noise Matched averages, opposite reactions to the same case features

Once an audit surfaces scatter, the authors split its sources. Level noise is the simple kind: some judges sit higher or lower across every case. A sentencer whose average runs above peers on equivalent crimes adds level noise.

Pattern noise is subtler and, in the insurance and sentencing decompositions, typically accounts for far more total noise. Two judges can share the same overall average yet reverse each other case by case—one soft on first offenses and hard on repeats, the other the opposite.

Average their averages and nothing looks wrong. Compare defendant by defendant and inconsistency is huge. Ordinary supervision almost never catches it, because each live case goes to one person alone.

Remedies aimed only at consistently harsh or lenient outliers therefore miss most of the problem. Later fixes in the book target how features get weighed, not merely where a global dial sits.

In practice: Two judges with matching average sentences still create a lottery if one is lenient on first offenses while the other is tough on them and softer on repeats.

4

When you disagree with yourself

Hiring better people cannot lock in reliability, because even one skilled professional shifts with mood, fatigue, and the order information arrives.

A third layer is noise inside a single person across occasions. The same doctor, judge, or underwriter given an identical case on two days often produces two answers though nothing in the file changed.

Fingerprint examiners re-shown a print they earlier called a match sometimes reverse the call when they do not recognize the repeat, especially once irrelevant context tags along. Order effects work similarly: material seen first anchors harder than the same material seen later, so two readers of one personnel file can diverge from sequence alone.

Some headline field findings on incidental schedule factors are contested; the authors still treat mood, fatigue, and context leaking into judgment as well supported by controlled experiments.

Selection and training therefore cannot finish the job. An excellent professional remains a slightly different instrument tomorrow morning, which pushes the argument toward process design over better headcount.

In practice: Fingerprint examiners reverse prior match calls on unmarked repeats when surrounding context changes, even though the print itself is unchanged.

5

Groups amplify scatter

Putting qualified people in a room does not average out idiosyncrasies; cascades and polarization often yield a more extreme, falsely confident shared verdict.

Replacing solo calls with group talk looks like an obvious fix. The authors argue it frequently makes noise worse.

Informational cascades do early damage. Whoever speaks first pulls later speakers toward that opening view, so the outcome tracks seniority or volume more than independent judgment and then dresses up as consensus.

Group polarization compounds it. People who already lean one way typically finish more extreme after discussion than the average of their private priors. A jury mildly inclined toward harshness often grows harsher, not milder.

Genuine averaging requires independence first: private, simultaneous estimates combined statistically. A show of hands after open talk is not that. The practical line is structured collection before any discussion.

In practice: A committee’s shared verdict often mirrors the first senior voice in the room rather than the mean of uninfluenced private views.

6

The illusion of agreement

Noise persists because judgment never feels arbitrary from the inside, so each professional assumes a colleague seeing the same facts would reach essentially the same conclusion.

If scatter is this large, why do organizations tolerate it unnoticed? Workflow is half the answer: each file usually goes to one person, so side-by-side disagreement never appears.

Psychology supplies the rest. The authors name the illusion of agreement. A diagnosis or hiring impression does not feel like one draw among many plausible ones; it feels like seeing what is there. The judge therefore never tests the peer match quietly assumed.

That confidence links to overconfidence patterns the authors invoke: people poorly sense how much history, mood, and values shape a call, and judgment-heavy fields rarely score outcomes against a clear right answer that would force the lesson.

Without feedback—what another judge would have sentenced, what a rejected applicant would have done—noise lasts until someone runs a deliberate audit. Measurement, not intention, is the required first step.

In practice: A loan officer almost never learns how a rejected applicant would have performed, so nothing challenges the sense that the call was simply the facts speaking.

7

Decision hygiene practices

Breaking a complex call into independently scored sub-assessments before any overall verdict removes much noise from first impressions and order effects without erasing expertise.

The practical program is decision hygiene, borrowed from the public-health idea of handwashing: steps that cut error without naming the specific disaster avoided in any one case.

Core moves include splitting judgment into separate sub-assessments recorded before a global conclusion forms; sequencing information so subjective or overall impressions arrive last; and aggregating multiple independent estimates statistically rather than through open discussion.

The mediating assessments protocol makes this concrete for interviews and evaluations. Raters score several specific attributes one at a time on scales anchored to concrete examples, and only after every sub-rating is locked do they form an overall judgment. Structured hiring interviews and formal grant reviews are working versions of the same idea.

None of this removes the need for skill. It aims to stop irrelevant order, mood, and early global impressions from driving variability, while leaving genuine expertise intact on well-defined sub-questions.

In practice: In a hiring process using mediating assessments, interviewers finish every trait rating on anchored scales before they may record an overall candidate score.

8

Formulas without bad days

Even crude statistical rules often outperform unaided experts on prediction tasks because they never tire or play favorites, yet people distrust algorithms and punish their errors more than identical human ones.

Human expert Real skill plus bad days, order effects, and one feature hijacking the rest

Simple rule Fixed weights on a few variables—same answer every time, no occasion noise

One strong path is replacing unaided judgment with a rule or formula. The evidence line runs through Paul Meehl’s 1954 review: in the great majority of roughly one hundred studies, simple statistical formulas matched or beat clinical experts on parole violation, academic success, and psychiatric diagnosis.

A mechanical rule carries zero occasion noise and zero pattern noise among judges, because there is only one consistent judge. Even equal weights on a handful of relevant variables tend to win—not from deeper insight, but from never tiring and never letting one salient feature overrun the rest.

Resistance stays strong. The book describes algorithm aversion: people still prefer the human after seeing the formula is more accurate, and they judge a formula’s errors more harshly than the same error from a person.

The authors treat the pushback as understandable—wanting individualized consideration, unease delegating parole or treatment to a machine, fear that biased training data will scale bias. Their narrower claim is that documented accuracy gains mean resistance to using rules even as one input deserves harder scrutiny than it usually receives.

In practice: Meehl’s review found simple formulas matched or beat expert clinical judgment in the great majority of about one hundred prediction studies across parole, grades, and diagnosis.

9

Consistency is not fairness

Driving noise out leaves underlying bias intact and can sacrifice local knowledge, mercy, adaptability, and the legitimacy some decisions draw from visibly human judgment.

Noise reduced Like cases treated alike; less of a personnel lottery

Still at stake Underlying bias, lost mercy, and irregular cases no box captures

The authors refuse to sell noise reduction as free. Heavy structure and rules can flatten the adaptability that helps with unusual cases—the applicant who fits no checklist box yet is still a sound risk.

Excessive standardization can demoralize skilled professionals who feel procedure is replacing craft, and can dull the independent noticing that sometimes catches what a formula would miss.

Consistency is also not fairness. An identical rule can still be a bad rule; removing scatter from a biased system only applies the bias more evenly. Noise work and bias work remain separate projects.

Some domains may want limited noise on purpose. In sentencing, individualized mercy can be part of what justice is supposed to offer, and not every judicial disagreement is pure error rather than a values clash. The authors’ answer is deliberate weighing by domain: killing order effects is nearly free; binding algorithms need a real stakes-and-values conversation.

In practice: A rigid loan checklist yields consistent yes-or-no decisions while missing the unusual applicant whose circumstances fit no box but who would still repay.

Key terms Noise Unwanted variability among judgments that should agree on the same case Bias Systematic tilt—the average of many judgments misses in one consistent direction Noise audit Giving identical real cases to many judges independently and measuring the spread of answers Level noise Stable differences in how high or low judges sit across the board Pattern noise Idiosyncratic responses to particular case features even when overall averages match Occasion noise The same person giving different answers to the identical case across times or contexts Illusion of agreement The untested assumption that a colleague seeing the same facts would reach essentially your conclusion Decision hygiene Structured practices that reduce scatter—independent sub-scores, deliberate sequencing, statistical aggregation Mediating assessments protocol Rating defined attributes separately on anchored scales before any overall verdict is formed Algorithm aversion Preferring human judgment and punishing formula errors more harshly even after the formula proves more accurate

What to do

Run a small noise audit on one recurring decision by giving three or four people the same real case independently and comparing answers before any discussion. Collect private judgments in writing first, then aggregate them; do not average opinions formed after open talk. Break your next evaluation into three or four specific sub-questions with defined scales, score each separately, and only then record an overall verdict. Where a decision repeats and enough data exist, test whether a simple checklist or weighted formula beats your current ad hoc approach.

Questions to think with Where does your team assign the same kind of case to different people without ever comparing their verdicts side by side? Which high-stakes judgments could you split into independent sub-scores before anyone forms a global impression? If a simple rule matched your experts’ average accuracy with less scatter, what values would still make you keep unaided human judgment? How would you separate pattern noise from defensible differences in professional values inside your domain? What would a noise audit of your last ten parallel decisions likely reveal that intuition currently hides?

The other side

The fairest criticism is that the strongest evidence sits in domains—underwriting, sentencing, forecasting, personnel—where answers share a common scale, while politics, ethics, art, and long-term strategy may not transfer cleanly. Some vivid occasion-noise field examples have been challenged as possible artifacts of case ordering. The book also leans on an earlier heuristics-and-biases literature that has faced replication trouble, offers little practical guidance for the internal politics of stripping discretion from senior holistic judges, and undersells how much judgment is asked to do beyond prediction—mercy, visible human legitimacy, and discretion a formula cannot hold.

Who it's for

Executives and managers who depend on staff for consequential case-by-case calls will gain the most, along with professionals in law, medicine, insurance, and hiring who want to understand why their own verdicts swing. Readers of behavioral economics looking for the quieter twin of cognitive bias will find the full argument worth reading.

Read the full summary in Library →