Retrieval-augmented generation looks like one feature and behaves like a small system. Between the embedding, the retrieval, the context assembly, and the generation, a single RAG query touches several cost drivers, and most teams track only the last one. Tracking RAG cost means tracking the whole pipeline.
The cost drivers in a RAG query
A RAG query has four cost stages. Embedding the query, retrieving from the vector store, assembling the retrieved context, and generating the answer. The generation call is the one everyone sees, but the retrieved context is what drives the token count, and the token count is what drives the generation cost.
Retrieved context is the hidden driver
The biggest lever in RAG cost is how much context you retrieve and stuff into the generation call. Retrieve generously and every query carries thousands of input tokens; retrieve precisely and the same answer costs a fraction as much. Track the average input tokens per RAG query, because that number, more than the model choice, determines the cost.
Track embedding and retrieval cost too
Embedding has its own per-token cost, and vector search has its own infrastructure cost that scales with index size and query volume. On high-traffic RAG features these are not rounding errors. Track them alongside the generation cost so the full per-query cost is visible, not just the model call.
Track cost per answered query
The unit that matters is cost per useful answer. Group the embedding, retrieval, and generation cost for one query and price the answer. That per-query number is what lets you compare a precise, cheap RAG design against a generous, expensive one that produces the same answer quality.
Attribute RAG cost to the feature
Tag each RAG query with the feature and team behind it, so retrieval-heavy features are attributed accurately. A search feature and a support feature may share a RAG pipeline but have very different query volumes and context sizes, and attribution keeps their costs distinct.
RAG is a pipeline, not a call. Track context size, embedding, retrieval, and generation together as cost per answered query, and you will see that the largest lever is usually how much context you retrieve, not which model you generate with.