5 AI agent performance metrics to pair with token cost
The vanity metric trap
Token consumption is an important cost input, but it cannot establish output quality by itself.
These five metrics can help assess whether an automation is working. Their definitions, collection windows, and targets must be calibrated to the workflow and its risk.
1. Task Completion Rate
What percentage of tasks finish successfully? Not "the API returned 200" — did the task actually complete? If a 10-step workflow fails at step 7, that's a 0% completion rate, not 70%.
Illustrative starting SLO: Define the acceptable completion rate from the workflow's baseline and consequence of failure; do not treat 95% as universal.
2. Mean Time to Recovery (MTTR)
When something fails, how long does the system take to recover? Recovery may be automatic, operator-assisted, or manual depending on the configured policy and the safety of retrying.
Illustrative starting SLO: Set separate recovery objectives for safe retries, human escalation, and failures that require rollback.
3. Coordination Efficiency
How much work is duplicated? If two agents both analyze the same data independently, that's wasted compute. High coordination efficiency means agents share context and build on each other's work.
Illustrative starting SLO: Establish a measured duplication baseline, then reduce unnecessary repeated work without removing independent review.
4. Output Quality Score
This is the hardest metric to automate, but the most important. Is the output actually good? Does it solve the problem? Does it follow the format you specified?
For Samsarix, this is partially captured by our Focus metric — are agents working on the right things?
Illustrative starting SLO: Define a reviewed rubric and sample size before setting an acceptable correction rate.
5. Cost Per Completed Task
Not cost per token. Cost per completed task. If a task requires 100,000 tokens but fails 5 times before succeeding, the real cost is 500,000 tokens. If another approach uses 200,000 tokens and succeeds first time, it's 2.5x cheaper despite higher per-attempt cost.
Illustrative starting SLO: Hold task definition and quality constant, then compare cost per accepted result over a meaningful window.
What to stop tracking
- Token counts alone — incomplete without completion and quality context
- API response time alone — incomplete without end-to-end task time
- Number of agents used — more agents isn't better, better coordination is
The bottom line
If you're measuring AI success only by token consumption, you're optimizing one input instead of the outcome. Track completion, recovery, efficiency, reviewed quality, and real cost per accepted task. Together, those measures provide stronger operating evidence than any one metric alone.
Track metrics that matter. Start free → UCF metrics, task tracking, and real cost analytics included.