Training gets the headlines, but inference is where the money actually goes for most organizations running AI in production. Every user interaction is an inference call, and those calls accumulate into the bulk of the bill. Tracking inference costs precisely is what keeps a live product's economics under control.
The two shapes of inference cost
Inference cost comes in two forms depending on how you run models. With hosted APIs, you pay per token, so cost tracks input and output tokens per call. With self-hosted models, you pay for the GPUs whether they are busy or idle, so cost tracks utilization and throughput. The two require different tracking, and many organizations run both at once.
For API inference, track tokens both ways
You pay for input tokens as well as output tokens, and on many workloads input dominates. Track the input-to-output ratio per feature. A feature sending large contexts to produce short answers is spending most of its inference budget on input, which points directly at prompt and context optimization.
For self-hosted inference, track utilization
Self-hosted inference cost is dominated by whether your expensive hardware is being used. Track GPU utilization and requests per GPU-hour. Low utilization means you are paying for idle capacity, and the fix is batching, autoscaling, or consolidating workloads, not more hardware.
Track cost per inference outcome
Whichever way you run models, the number that matters is cost per useful result, not cost per call. Group the inference calls that go into one completed task and price the task. An agentic feature that makes twelve inference calls per task has a very different cost profile from a single-call feature, and only the per-outcome view makes that visible.
Attribute inference to features and teams
Inference cost arrives as an aggregate unless you tag it. Attach a feature and team identifier to each inference call so the cost breaks down by what drove it. Attribution is what turns a large inference bill into a set of decisions about which features and teams are worth their inference budget.
Inference is the recurring cost of AI in production. Track tokens for API inference, utilization for self-hosted, and cost per outcome for both, and the largest line on your AI bill becomes one you can actually manage.