What does it mean that calibration requires feedback?

You cannot improve the alignment between your confidence and your accuracy without external data that reveals the gap between what you believed and what actually happened. Calibration without feedback is guesswork about guesswork.

Example: A product manager estimates a 90% probability that a new feature will increase user retention. She ships it. Retention drops. Without tracking this prediction against the outcome — without feedback — she would never update her mental model. She would continue shipping features with the same misplaced certainty, accumulating a pattern of confident failure invisible to her own introspection. But she keeps a prediction log. After six months and forty tracked predictions, she discovers that events she rated at 90% confidence actually occurred only 60% of the time. The gap is enormous and would have been invisible without the log. She recalibrates. Over the next six months, her 90% predictions start landing at 82%. Still imperfect. But the feedback loop is compressing the gap — not through willpower or intelligence, but through the simple act of comparing what she believed to what happened.

Try this: Start a calibration journal. For seven consecutive days, make five predictions each day about events whose outcomes you will know within 48 hours — project deadlines, meeting outcomes, whether someone will respond to your email, weather, traffic, anything with a verifiable result. For each prediction, assign a confidence level: 50%, 60%, 70%, 80%, 90%, or 95%. At the end of the week you will have 35 predictions. Tally your results by confidence bucket. If your 80% predictions came true 80% of the time, you are well calibrated at that level. If they came true only 55% of the time, you have discovered an overconfidence gap that introspection alone would never have revealed. This is not an exercise in humility. It is an exercise in measurement.