Why does agent reliability metrics fail?

The most common reason agent reliability metrics fails: Treating reliability as a binary — the agent either 'works' or 'doesn't work.' This collapses a rich, multi-dimensional signal into a useless bit. An agent with 95% reliability and a 30% false-fire rate has a completely different failure profile than an agent with 70% reliability and a 0% false-fire rate. The first fires almost every time it should but also fires when it should not — a sensitivity-heavy, specificity-poor agent. The second misses nearly a third of its triggers but never fires incorrectly — a conservative, specificity-heavy agent. These two agents need opposite interventions. Treating both as simply 'unreliable' leads you to apply the wrong fix and wonder why nothing improves.

The fix: Select three cognitive agents you rely on regularly — your daily planning agent, your emotional regulation agent during conflict, your focused-work agent, your active-listening agent, or any others you have identified in earlier phases. For each agent, define: (1) The trigger condition — what situation should activate it? (2) The observation window — the past 14 days. (3) The hit count — how many times the trigger occurred and the agent fired correctly. (4) The miss count — how many times the trigger occurred and the agent failed to fire. (5) The false-fire count — how many times the agent fired when the trigger condition was not actually met. Calculate reliability rate (hits / total triggers) and false-fire rate (false fires / non-trigger occasions) for each. You now have six numbers that describe the reliability profile of three core agents. Write them down. This is the beginning of your monitoring dashboard's reliability layer.

The underlying principle is straightforward: Track how often each agent fires when it should and does not fire when it should not.