Before finalizing a schema update, verify the new version handles five cases where the old one worked
Before finalizing any schema update, list five situations where the old schema produced accurate results and verify the new schema handles all five to ensure backwards compatibility.
Why This Is a Rule
Schema updates that fix one failure while breaking previously working cases produce net regression — you've traded one error for several. This is the epistemic equivalent of a software regression: the fix for one bug introduces three new ones because the test suite didn't cover the existing functionality.
The backwards-compatibility check lists five situations where the old schema produced accurate results and verifies the new schema handles all five. If the new schema fails on any case the old one got right, the update is a regression — it needs further refinement before deployment.
Five cases is the minimum viable regression test. Fewer than five might miss important success patterns. More than five adds diminishing returns for the effort. The cases should span the old schema's operational range: different contexts, different scales, different actors.
When This Fires
- Before finalizing any schema revision triggered by disconfirming evidence
- After generating a new version of a belief or operating rule
- During schema evolution when the new version feels better but hasn't been regression-tested
- Any time you're replacing one mental model with another
Common Failure Mode
Testing the new schema only against the case that broke the old one: "The new schema explains the failure, so it's better." But does it also explain the five successes? A schema that explains one failure while failing to explain five successes is a net loss. The new schema must be a superset — handling everything the old one handled plus the new case.
The Protocol
Before finalizing a schema update: (1) List five specific situations where the old schema produced accurate, useful results. Choose diverse cases across the schema's range. (2) For each, verify: does the new schema also handle this case correctly? Does it produce the same (accurate) prediction or recommendation? (3) If all five pass → the new schema is backwards-compatible. It handles the new case AND preserves old successes. Finalize the update. (4) If any case fails → the update introduces a regression. Refine the new schema until it handles both the new case and all five old cases.