Same three symptoms before three failures = name the pattern and automate the alert
When the same symptom triad precedes system failures across three independent incidents, document it as a named detection pattern and build an automated alert triggered by that specific combination.
Why This Is a Rule
System failures rarely announce themselves with a single clear signal. They announce themselves with combinations — a symptom triad (three co-occurring signals) that individually seem innocuous but together predict failure. Memory response time increases slightly AND error rate ticks up 2% AND a specific log message appears with unusual frequency. Each symptom alone is within normal variance. Together, they're a reliable failure predictor.
When the same triad precedes failures across three independent incidents, you have a validated detection pattern — a reliable leading indicator of failure. This is too valuable to leave as tribal knowledge in one engineer's head. It needs two things: a name (so the team can reference it: "We're seeing a Pattern-7 forming") and an automated alert (because human detection of multi-signal combinations under operational pressure is unreliable).
Three independent incidents is the validation threshold because a pattern observed once could be coincidence, twice could be correlated by context, but three times across independent incidents confirms a genuine structural relationship between the symptoms and the failure.
When This Fires
- After the third incident where the same pre-failure symptoms appeared
- During postmortem analysis when reviewing incident timelines for precursor signals
- When building or improving monitoring and alerting systems
- Any context where early warning patterns have been observed but not formalized
Common Failure Mode
Documenting the pattern but not automating the alert. The documented pattern lives in a postmortem report that no one reads during the next incident. Automation is the point — the pattern fires automatically, alerting the team before the failure completes, giving them time to intervene.
The Protocol
When the same symptom combination precedes a third independent failure: (1) Document the pattern: name it, describe the three symptoms, note the typical time gap between symptom onset and failure. (2) Name it concisely: "memory-spike-error-log triad" or "Pattern-7: slow-query cascade." (3) Build an automated alert that triggers when all three symptoms co-occur within the identified time window. (4) Share the named pattern with the team so it enters the collective monitoring vocabulary.