After 15-20 judgments, check if your errors are bias or noise

Why This Is a Rule

Kahneman, Sibony, and Sunstein's Noise (2021) establishes a critical distinction: bias is systematic error in one direction (you always overestimate project timelines by 30%), while noise is random variation (your estimates scatter unpredictably). The distinction matters because the fixes are completely different.

Bias requires a correction factor: if you're consistently 30% over, apply a 0.7 multiplier. Noise requires aggregation: average multiple independent estimates, or use structured protocols that reduce variability. Applying the wrong fix makes things worse — correcting for bias when the problem is noise adds systematic error to random error.

You need 15-20 data points to distinguish the two. Below that threshold, a directional cluster might be noise, and apparent scatter might mask a real bias. At 15-20 judgments, the pattern becomes statistically readable: do your errors point consistently in one direction, or do they spray in all directions?

When This Fires

• You've made enough project estimates to check your track record
• You've accumulated hiring decisions, performance predictions, or risk assessments in one domain
• You notice your judgments seem "off" but can't tell if you're biased or just inconsistent
• Any domain where you make repeated judgments of the same type

Common Failure Mode

Treating all errors as bias. You overestimated three projects in a row and conclude "I'm always too optimistic." But three data points isn't enough — and the next three might be underestimates. Without enough data to distinguish bias from noise, you apply a correction factor that makes half your estimates worse while improving the other half.

The Protocol

After 15-20 judgments in the same domain: (1) List each judgment alongside the actual outcome. (2) Calculate the error (judgment minus actual) for each. (3) Check directionality: do errors cluster positive (overestimate) or negative (underestimate), or do they scatter both ways? (4) If directional → calculate the average error and apply it as a correction factor to future judgments. If scattered → use aggregation strategies: multiple independent estimates, structured frameworks, or checklists that reduce noise.