Direct answer: First-try success rate. It is the share of tasks a workflow completes correctly on the first attempt, with no retry, re-prompt, or discarded result. Cost per call goes vain the moment a task fails and reruns three times, because you paid the low price three times over. It tells you whether your spend is buying work or buying attempts.
Cost per call is the metric every AI dashboard leads with, and it is the one that fools the most people. It is easy to compute, it looks rigorous, and it sits right next to the invoice, so it feels like the number in charge. I have sat with leaders who cut their cost per call by moving to a cheaper model and then watched their total bill hold steady or rise, and every time the explanation was the same. The cheaper calls were failing more often, so the team ran more of them. The metric on the dashboard went down while the thing it was supposed to represent went up.
The metric that actually governs the outcome is first-try success rate, and the argument for it is mechanical, not philosophical. A workflow exists to complete a task. If it completes on the first attempt, you pay once. If it fails and the engineer re-prompts or the agent retries, you pay again, and the second and third attempts cost real money that lands in the same line item as the first. Cost per call measures the price of one attempt. First-try success measures how many attempts a result actually took. One of those numbers can be halved while your spend doubles. This post is about why first-try success dominates cost, and how to measure it across your workflows without fooling yourself.
Why cost per call is a vanity metric
A vanity metric is one that moves in a way you can celebrate while the underlying reality moves against you, and cost per call fits the definition exactly. Its flaw is that it treats every call as an independent event with its own price, when in production calls come in chains. An engineer prompts, the answer is wrong, they prompt again with a correction, they get something usable on the third try. Three calls, one task. Cost per call reports the average price of those three calls and calls it efficiency. The honest number is the cost of all three divided by the one task that got done, which is three times higher.
Agents make this worse because the retries are automatic and invisible. A coding agent that fails a task does not stop and ask. It tries a different approach, calls a tool again, resends context, and keeps going until it succeeds or gives up. Every one of those internal attempts is a billed call, and every one lowers your average cost per call while raising your cost per completed task. Optimizing cost per call in an agentic workflow can actively push you toward cheaper, weaker models that fail more and retry more, which is the opposite of what you wanted.
What first-try success actually measures, and what it does not
First-try success is the fraction of task attempts that reach a correct result on the first call. To measure it you need two things, a clear boundary for what one task is, and a way to know whether the result was correct. The task boundary is a definition you set per workflow. For a code-generation workflow, one task might be one requested change. For an extraction workflow, one task is one document processed. Getting this boundary right is most of the work, because a task defined too loosely hides retries inside it and a task defined too tightly counts normal multi-step work as failure.
Here is the honesty rail that has to sit at the center of this metric, because without it the number lies. Completion status is not answer quality. When you capture that a call finished, you know the request returned, not that the content was right. A model can return a confident, well-formed, completely wrong answer with a clean success status. So first-try success measured purely from status semantics tells you the call did not error, which is a weaker claim than the task landed. Be explicit about which one you are reporting. Status-based first-try success is a useful floor, it catches the failures that surface as errors, timeouts, and rejections. Quality-based first-try success requires a check against a known-good answer or a rubric, and it is the stronger number. When you publish a first-try success rate, say which of the two it is, because conflating them is how a metric meant to fight vanity becomes vain itself.
How to measure first-try success across workflows
Start where the signal is cheapest to get, which is the status layer. Instrument each workflow so you can see, per task, whether the first call completed without an error, a timeout, a content rejection, or an immediate re-run by the same engineer or agent within a short window. A re-run inside a minute on the same inputs is a strong signal the first attempt failed even if it returned a success status. This gives you a status-based first-try rate per workflow with no rubric required, and it is enough to rank your workflows from healthiest to most retry-heavy.
Then add quality checks where the task allows automated scoring. Extraction, classification, and structured generation can be checked against known-good answers with an eval harness, the same harness you would use to compare models. For those workflows you get a true quality-based first-try rate. For open-ended workflows where automated scoring is hard, sample. Pull a fixed number of first attempts each week and have a human or a scoring model judge correctness against a rubric, and track the sampled rate over time. You do not need every task judged to trust the trend, you need a consistent sample.
Watch the metric per workflow, not as one org-wide average, for the same reason cost per engineer misleads. Workflows have different achievable ceilings. A routing classifier should sit near ninety-five percent first-try success and a hard open-ended reasoning task might top out far lower, and both can be healthy. The number you act on is each workflow against its own history. When a workflow's first-try rate drops, something changed, a model update, a prompt regression, a shift in inputs, and you can catch it in real time instead of discovering it as a cost bump on the month-end bill. Cost is the lagging shadow of this metric. First-try success is the thing casting it.
Oberhahn derives first-try success and retry patterns per workflow from the telemetry your AI tools already emit, keeping status-based and quality-based signals distinct so you know which you are reading, and you can see it running on sample data in a demo before connecting a source.
Frequently Asked Questions
What AI metric matters more than cost?
First-try success rate, the share of tasks a workflow completes correctly on the first attempt without a retry, re-prompt, or discarded result. Cost per call is misleading because a cheap call that fails and reruns three times costs you three times over while the per-call number looks low. First-try success tells you whether spend is buying completed work or repeated attempts.
Why is cost per call a vanity metric?
Because production calls come in chains, not isolated events. One task can take three calls when the first two fail, and cost per call averages those three into a low-looking number while the true cost per completed task is three times higher. In agentic workflows the failed attempts are automatic and invisible, so optimizing cost per call can push you toward weaker models that retry more.
Does a completed API call mean the answer was correct?
No. Completion status tells you the request returned without an error, not that the content was right. A model can return a clean success status with a wrong answer. Status-based first-try success is a useful floor that catches errors and timeouts. For a true quality measure you need to check the result against a known-good answer or a rubric, and you should always say which of the two you are reporting.
How do I measure first-try success for open-ended tasks?
Sample rather than score everything. Pull a fixed number of first attempts each week and have a human or a scoring model judge correctness against a consistent rubric, then track the sampled rate per workflow over time. You do not need every task judged to trust the trend, only a steady sample and a stable definition of what one task is.