Organizations that self-host models often assume they have escaped AI cost tracking, because there is no per-token API bill to watch. In reality the cost just moved, from tokens to infrastructure, and it is harder to see. Tracking self-hosted AI cost means tracking the hardware, not the calls.
The cost is the hardware, busy or idle
With a hosted API you pay only for what you use. With self-hosted models you pay for the GPUs whether they are serving requests or sitting idle. That changes the entire tracking question from cost per call to cost per hour of capacity, and idle capacity is pure waste that a token-based mindset never accounts for.
Track utilization first
The most important self-hosted metric is GPU utilization. Low utilization means you are paying for capacity you are not using. Track requests per GPU-hour and average utilization across your fleet, because improving utilization through batching or autoscaling is usually a bigger lever than any model-level optimization.
Derive an effective cost per request
To compare self-hosted against hosted alternatives, translate infrastructure cost into an effective cost per request: total GPU cost for a period divided by requests served. This is the number that tells you whether self-hosting is actually cheaper than an API at your current volume, and the answer changes as utilization and volume change.
Attribute shared infrastructure
Self-hosted inference is usually shared across teams and features. Track which teams and features drive the load, and allocate the infrastructure cost by their share of requests. Without allocation, a shared GPU cluster looks like an unowned central cost, and no team feels responsible for the load they create.
Track the full cost, not just compute
Self-hosting adds costs a token bill does not: the engineering time to run the serving stack, the overprovisioning for peak load, and the tooling around it. Track these alongside the raw GPU cost so the comparison against a hosted API is honest, because the sticker price of a GPU is not the full cost of self-hosting.
Self-hosting does not remove AI cost tracking; it changes what you track. Watch utilization, derive an effective cost per request, allocate shared infrastructure, and count the full operational cost, and you will know whether running your own models is actually the cheaper path.