Managing AI Token Costs with Azure API Management

I attended my first Integrate recently, and after hearing colleagues talk about it for years, I went in expecting relevant content and came away with more than I bargained for. If one theme ran through every track, it was artificial intelligence, not just sessions dedicated to it, but AI reshaping how we think about integration itself. Long before the conference started, one session had already made its way onto my must-watch list: Managing AI Token Costs with Azure API Management, delivered by Alex Vieira, a Global Black Belt at Microsoft who works with strategic customers on AI adoption.

Alex did not disappoint us. He has a knack for taking a genuinely new problem and grounding it in practices teams already understand. In this case, the problem is that AI workloads are priced and consumed very differently from the APIs most of us have spent years governing. Rather than reinventing cost governance from scratch, Azure API Management extends the FinOps thinking most organisations already apply to cloud spend into the world of tokens and models.

This post is a recap of what he covered, along with what Valorem Reply is taking from it.

Six reasons AI spending is hard to manage

Alex opened with the challenges that make AI cost management genuinely different from what IT teams are used to:

1. Tokens are a confusing unit. Prompt tokens and completion tokens are different things, priced differently, and counted differently depending on the model and provider.

2. GPUs are scarce. Most inference runs on GPUs; demand is high, and that keeps prices volatile.

3. Attribution is messy. Models get shared across users, agents, tenants, and departments. Working out who used what is a real problem for most organisations right now.

4. Forecasting is hard. Moving from pilot to production means predicting cost across many AI use cases, and there is not much history to base it on.

5. Models keep changing. New ones arrive weekly. Some are cheaper nano and mini variants; some are pro models with reasoning or multimodal features. Picking the best model for the job is a constant trade-off.

6. AI adoption is not just IT anymore. Multiple departments adopt AI on their own, sometimes as shadow IT, which makes a company-wide view even harder.

One useful piece of grounding: model providers charge by tokens; token counting varies by provider, but every response tells you exactly how many tokens it consumed. That number is the raw material for everything else in this post.

Start with FinOps, then go deeper into AI 

Rather than inventing something new, this approach anchors everything to the FinOps Foundation framework, which now includes a dedicated scope for AI. The idea is simple: align with the industry practices that Microsoft and other vendors already follow, then drill into the AI-specific parts.

Microsoft Foundry lets you deploy models from a large catalogue and monitor cost per model, including input and output tokens over time. Azure Cost Management then shows what you were charged, and the bill arrives at the end of the month like any other Azure service.

Here is the gap, though. Those views tell you what a model deployment costs in total. They cannot tell you that the customer support bot used 60 per cent of it, and a side project in marketing used the rest. When one model serves many applications, departments, and users, you need something sitting in the middle. That is the AI gateway.

The AI gateway: one place to watch and control usage

Azure API Management acts as the gateway between your model consumers (apps, services, agents, even tools like GitHub Copilot) and the model providers behind them. Once traffic flows through that single point, two things become possible: you can limit usage, and you can attribute it.

Limits and quotas

The token rate limiting policy was announced at INTEGRATE two years ago, and it has matured a lot since. With a single policy, you can assign token limits and quotas to each consumer of a shared model. A consumer that exceeds its allowance gets rejected at the gateway before the request ever reaches the model, so you are not paying for it, and one noisy app cannot drain the quota everyone else depends on.

Coverage has expanded beyond Azure OpenAI. The same policies now work across the Foundry model catalogue and direct providers, including OpenAI, Anthropic, Google, AWS, and Hugging Face.

Logging and attribution

The part of the demo that impressed me most was how little configuration this needs now. On any inference to API in APIM, you open the Azure Monitor tab and enable LLM message logging. There is a meaningful choice inside that toggle: log the metrics only (token counts per request) or also log the prompts and completions themselves.

Once enabled, everything lands in Azure Monitor tables where you can query it and filter by model name, prompt tokens, completion tokens, subscription ID, and even the individual user, which the gateway can extract from the JWT token sent with each request. Alex showed a custom pricing table joined onto those logs to compute actual money spent per user, not just tokens. From there, you can export to CSV or Power BI or build dashboards and workbooks. He also mentioned, honestly enough, that a coding assistant is a major help in writing the KQL queries.

The logs also work with streaming responses, chunk by chunk, with the exact token count confirmed at the end. And since Build, the metrics distinguish reasoning tokens and cached tokens separately, which matters because cached input tokens are cheaper, and reasoning models consume tokens you would otherwise miss.

Alerts that act, not just notify

Dashboards are only half the story. Azure Monitor alerts can be built on those same queries and wired to a Logic App that automatically suspends an APIM subscription when it exceeds its monthly quota. With thousands of Logic App connectors available, the response could just as easily be an email to the team lead or a ticket in your service desk. The point is that cost control can be automated, not just seen.

Alex also demoed an open-source FinOps dashboard built on these same APIs and Azure Monitor data, with drilldowns by user, subscription, and cost, plus an evaluation framework for comparing models against real outputs. It is deliberately open source so teams can tailor it, and the whole lab setup is available in the AI-Gateway samples repo on GitHub to clone and experiment with.

Spending fewer tokens in the first place 

Controls and visibility cap the damage, but the gateway can also directly reduce consumption. The main pattern here is semantic caching: incoming prompts are compared against previously stored completions using embeddings and vector similarity, and if a close enough match exists, the cached answer is returned. No model calls, no tokens, faster response. It is more advanced to set up than plain caching, but for workloads with repetitive queries it pays for itself.

He closed by noting that this is just one feature of the AI gateway. Capabilities like model fallback, where traffic shifts to another model when one struggles, add reliability on top of the cost story.

Valorem Reply takeaways 

• The gap is in attribution, not billing. Azure already tells you what AI costs in total. What most organisations cannot answer is who spent it, and the gateway is currently the practical way to close that gap.

• Visibility can be free but think before logging everything. Token metrics are just a few numbers per request and barely register in Azure Monitor costs. Logging full prompts and completions is useful for evals later, but it can contain sensitive data, so treat that as a security decision, not just a FinOps one.

• Automated cost control. An alert that suspends an over-quota subscription automatically is a vastly different safety net than a dashboard someone checks on Fridays.

• Token metrics are catching up with how models work. Separate reasoning and cached token metrics mean your numbers finally reflect what reasoning models really consume.

• Customers are further behind on this than you would think. Most teams only get serious about inference costs once production scale makes the bill impossible to ignore. Starting earlier is cheaper.

Learn more 

Microsoft's documentation on AI gateway capabilities in Azure API Management covers the technical details behind everything above, and the open-source AI-Gateway labs on GitHub let you try the whole setup hands-on, including the FinOps dashboard shown in the session.

How Valorem Reply can help 

Token cost governance is exactly the kind of thing that is easy to postpone until it becomes a problem. At Valorem Reply, we work with organisations building their Azure integration and AI platforms to make sure this kind of visibility and control is baked into designs from day one, rather than retrofitted after an unexpectedly large invoice. Whether that is architecting an AI gateway in Azure API Management, setting up token-level attribution and FinOps dashboards, or simply helping a team work out sensible limits for their specific workloads, this is the kind of work we help clients tackle regularly. If any of this sounds like a challenge you are facing, we would be glad to talk it through.