Direct answer: Run the same task against Claude, GPT, and a smaller open model, then compare what each actually cost against how often each landed the task without a retry. Reputation and price per token are guesses. Measure real cost and real first-try success on your own work, and the best model becomes a fact you read, not an argument.
Every few weeks someone in an engineering org rewrites a config to switch model providers, and the reason they give is almost never a number. It is a benchmark someone posted, a launch thread, a price-per-token comparison, or a hunch that the newer model must be smarter. I have watched teams move their whole agent stack to a cheaper model, celebrate the lower unit price, and quietly spend more overall a month later. The switch was decided by reputation and sticker price, and both of those are proxies for the thing that actually matters, which is how the model performs on the specific job you are running.
So the frame I use for model choice is a bake-off, and the word matters. A bake-off is not a leaderboard someone else ran on someone else's tasks. It is the same task, your task, run against several models under conditions you control, scored on two axes you care about. The two axes are real cost and real first-try success. Everything in this post is about how to get honest numbers on both, because once you have them the decision makes itself.
Why reputation and price per token both lie
Price per token is the number vendors compete on because it is easy to print, and it is close to meaningless on its own. A model that costs half as much per token but needs three attempts to complete your task is more expensive than the pricey model that lands on the first try. The cheap-per-token model also tends to send more tokens, because weaker reasoning shows up as longer chains, more tool calls, and more back-and-forth. You pay the low rate on a bigger pile of tokens, several times over, and the line item still says you saved money.
Reputation lies in a different way. A model earns its reputation on public benchmarks and viral examples, and your job looks nothing like those. Your job is a specific refactor pattern, a specific extraction schema, a specific class of customer-support reply, run against your codebase and your data with your prompt scaffolding. A model that tops a coding benchmark can underperform on your particular repo because your repo has conventions the benchmark never saw. The only benchmark that predicts your results is the one built from your tasks.
Build a task set that looks like your actual work
Before you compare anything, you need a set of tasks that represent the job. This is the eval harness, and building it is the part teams skip and then regret. Pull twenty to fifty real examples of the task from your logs. Real prompts, real inputs, real expected outcomes. If the job is code review, use twenty pull requests you already reviewed by hand so you know the right answer. If the job is data extraction, use fifty documents where you already know what the correct extraction is.
The reason you need known-good answers is that first-try success has to be checked against something. For extraction and classification, the check can be exact-match or schema-match, fully automated. For open-ended work like code or prose, the check is a rubric a human or a scoring model applies consistently. The harness runs the same task set through each model, records the output, and scores each output against the known answer. This is ordinary evaluation engineering, the same discipline you would apply to any system you were choosing between, applied to a model instead of a database.
One honesty note that governs everything downstream. A completed call is not the same as a correct answer. An API can return a clean two-hundred status, the request can finish, and the content can still be wrong. First-try success in a bake-off means the task landed correctly, judged against your known-good answer, not that the call returned without an error. Keep those two ideas separate or your whole comparison quietly measures the wrong thing.
Measure real cost per successful task, not per call
Now run the harness and count tokens. Count input tokens and output tokens for every attempt, including the retries, because the retries are where the real cost hides. Most provider SDKs return token counts in the response, and if you route through a proxy like LiteLLM you can capture them centrally without instrumenting each call site. Multiply by each model's rate to get cost per attempt, then sum every attempt a task needed to reach a correct result.
That sum, divided by the number of tasks that eventually succeeded, is cost per successful task. It is the number that actually decides the bake-off, and it reorders the field almost every time. The model with the lowest price per token frequently has the highest cost per successful task, because its cheap calls fail and rerun. The model that looked expensive often wins, because it lands on the first attempt and never triggers the retry loop. You cannot see any of this from a pricing page. You can only see it after the harness has run.
Include a smaller open model, because it sometimes wins
Always put a smaller open-weight model in the bake-off, something you can run on vLLM or SGLang against your own hardware. Not because open models win on hard reasoning, they usually do not, but because a large share of production tasks are not hard reasoning. They are classification, extraction, routing, and formatting, and a small model self-hosted can do those at a fraction of the cost with a first-try success rate close enough that the price difference dominates. The bake-off tells you which tasks are in that category and which are not, so you can route the easy work to the cheap local model and reserve Claude or GPT for the tasks that genuinely need them. That routing decision is worth more than any single model choice, and only the task-level numbers reveal it.
Run the bake-off again when a new model ships, and when your task mix changes. Model choice is not a one-time decision because the field moves and your workloads move with it. The harness makes rerunning cheap, which is the whole point of building it once.
Oberhahn captures the token counts, retries, and per-task attribution across every model your teams already call, so the cost-per-successful-task comparison assembles itself from live traffic instead of a one-off spreadsheet, and you can walk a demo on sample data before connecting a thing.
Frequently Asked Questions
How do I know which AI model is best for my job?
Run the same set of your real tasks against Claude, GPT, and a smaller open model, then compare cost per successful task and first-try success rate for each. The best model is the one that lands your specific work correctly at the lowest total cost including retries. Public benchmarks and price per token do not predict this, because your job is not the benchmark and cheap calls that fail and rerun are not cheap.
Is the cheapest model per token the cheapest to run?
Often no. A low price per token gets multiplied by more tokens and more retries when the model is weaker on your task, so cost per successful task can be higher than a pricier model that lands on the first attempt. Count every attempt a task needed, then divide by successful tasks, to see the real number.
Should I trust public model benchmarks for my decision?
Use them to pick which models to test, not to make the final call. Public benchmarks run on tasks that are not yours, against data and conventions your work does not share. Build a small task set from your own logs with known-good answers, and let that decide.
When should I use a small open model instead of a frontier model?
For classification, extraction, routing, and formatting, a smaller open-weight model on vLLM or SGLang often reaches a first-try success rate close to a frontier model at a fraction of the cost. Reserve Claude or GPT for tasks that genuinely need harder reasoning. A per-task bake-off shows you which category each workload falls into.