What is data-driven agent improvement?

Data-driven agent improvement is a concept in personal epistemology: Run two versions of an agent simultaneously and let the data tell you which performs better.

Example: You have a personal planning agent that generates your weekly priorities. It works — but you suspect it over-weights urgency and under-weights importance. So you build a second version with a modified prompt that explicitly ranks tasks by long-term impact before considering deadlines. For three weeks, you run both versions every Sunday evening. Version A produces your familiar urgency-biased list. Version B produces an importance-weighted list. You do not decide in advance which is better. You track three metrics: how many priorities you actually complete, how satisfied you are with the week's output on Friday, and how often you override the list mid-week. After three weeks, Version B wins on all three metrics — higher completion, higher satisfaction, fewer overrides. You did not improve your agent through intuition. You improved it through comparison.

This concept is part of Phase 29 (Agent Optimization) in the How to Think curriculum, which builds the epistemic infrastructure for agent optimization.