Die AgentinNICOLENDERS.COM
DEEN
← Dispatches

AnalysisTHE ARCHIVISTTHE TECHNICIAN

An AI gateway only sees half your costs

Published

A mechanical counting machine on a workbench with one stream of envelopes running through it while a larger stack bypasses it along a side rail into the same sack.AI generated
Manage costs effectivelyClaude Code Docs

The Azure invoice shows one line for Claude: CCU. One amount, no model, no team, no person. The per-model breakdown lives in the Monitoring tab of Microsoft Foundry, and even there it will not tell you who burned those tokens. This is not an oversight. It is the design.

I am currently looking at exactly this problem inside a large corporate environment, spread across three vendors: models in Microsoft Foundry, ChatGPT Enterprise, Claude. The obvious answer is an AI gateway, and the obvious candidate is LiteLLM. After two weeks of digging, my conclusion is not the one I expected.

A gateway only controls what runs on API keys

That sentence decides everything else. Sort your AI traffic into three categories before you talk about products.

One: application traffic. Your own apps, agents, batch jobs. They speak HTTP against an endpoint with a key. A gateway covers this completely. Virtual keys per team, hard budgets, fallbacks, logging.

Two: coding agents on API keys. Claude Code, Codex, OpenCode, when they authenticate against a key rather than a subscription. A gateway covers this too, with deductions.

Three: seat traffic. ChatGPT Enterprise, GitHub Copilot, Claude team seats. This traffic never reaches your endpoint. It is billed against an account you own, but the controls sit exclusively in the vendor's admin console.

OpenAI states this as plainly as anyone could wish: ChatGPT Business and Enterprise are separate from the API platform, and the subscription includes no API usage. A Codex user signing in with a ChatGPT account consumes workspace credits, the same user with an API key pays platform rates. One tool, two invoices, and only the second can go behind a gateway.

Read those three categories again with your own licence estate in mind. If the larger block of spend sits in category three, a gateway solves a smaller problem than the business case promises.

What LiteLLM actually solves

LiteLLM is at version 1.92.0, released 12 July 2026, and serves /chat/completions, /messages and /responses from one process. The real value is not cost tracking. You can get that elsewhere. The value is protocol translation: Claude Code speaks the Anthropic Messages API, Codex speaks the Responses API and nothing else, your own apps speak Chat Completions. LiteLLM is the only thing that accepts all three at once. Budgets are hard limits, set on key, user, team or organisation, and hitting one produces a refusal rather than a warning email.

Three reservations that rarely make it into the tutorials.

The first sits in LiteLLM's own documentation: budgets are evaluated against the database. With no database attached, max_budget fails open silently, with one warning at startup and nothing at request time. A gateway without Postgres is a router, not a spending limit.

The second is supply chain. On 24 March 2026 the PyPI releases 1.82.7 and 1.82.8 shipped a backdoor for roughly forty minutes that read OPENAI_API_KEY and ANTHROPIC_API_KEY straight out of the environment. The project points out that LiteLLM can land on a host as an unpinned transitive dependency of an agent framework. Verify the cosign signature, pin the version, filter egress.

The third is accountability. Anthropic links LiteLLM from its own gateway documentation and writes in the same breath that it neither operates nor audits nor endorses it. That is not a warning about the project. It is a statement about who carries the incident.

Microsoft now has a gateway too, with two front doors

The AI gateway in Azure API Management is not a separate product but a set of policies on the existing gateway. llm-token-limit caps per counter key, either as a TPM rate or as a quota over a period, answering with 429 or 403 respectively. llm-emit-token-metric writes token counts with custom dimensions into Application Insights, now including reasoning, cached and thinking tokens.

Two details decide whether it fits. The Anthropic Messages API is, per Learn, currently supported in the v2 tiers only. And the unified model API, which puts several backends behind one endpoint and translates formats, is in preview and OpenAI-compatible. It replaces LiteLLM for application traffic, not for Claude Code.

There is a second front door straight from Foundry, under Manage, also in preview. From there you set token limits and quotas per project without writing APIM policies. For an organisation already sitting on Azure, that is the path with the lowest operational load: managed identity instead of keys, telemetry in the Application Insights you already run.

The price is conceptual. APIM limits tokens, not money, and converting to currency is something you build yourself with custom dimensions and a workbook. And Azure budgets alert without stopping anything: Learn says explicitly that resources are unaffected and consumption continues, on a data latency of eight to twenty-four hours. An agent stuck in a loop overnight costs you a full working day before the number is even visible.

Claude in Foundry: good news with three footnotes

Claude reached general availability in Microsoft Foundry on 29 June 2026. Billing runs on Claude Consumption Units, metered hourly, invoiced monthly in arrears, and the CCU meter is MACC-eligible. If you hold an Azure commitment, Claude can draw against it. That is the strongest commercial reason to buy through Foundry rather than directly.

Visibility drops in exchange. One marketplace meter replaces the previous per-model meters, so Azure Cost Management shows you one line. The exact breakdown you need for chargeback leaves the invoice.

The region is missing. Azure-hosted Claude models come as Global Standard and Data Zone Standard, the latter for the US zone only. No EU data zone is documented at present. For regulated workloads in Europe that is where the conversation ends, before it gets to cost.

And the limits are lower than coding agents expect. Pay-as-you-go gives claude-opus-5 a default of 40 RPM, 40,000 uncached ITPM and 8,000 OTPM, shared across all deployments in the same subscription. Anthropic's own guidance for Claude Code at twenty to fifty users is 50,000 to 75,000 TPM per person. Do that arithmetic before you point ten developers at one Foundry deployment, and file the quota request early. Plan for it rather than discovering it.

The tools, side by side

Tool

Protocol

Route through a gateway

What you give up

Claude Code

Anthropic Messages

ANTHROPIC_BASE_URL plus ANTHROPIC_AUTH_TOKEN against /v1/messages

Beta headers that non-Anthropic backends reject with 400; model discovery, until /v1/models answers in Anthropic shape

Codex

Responses only

Its own [model_providers.x] block with wire_api = "responses"

Model metadata falls back to generic values, so context window and capability flags; openai, ollama and lmstudio are reserved provider IDs

OpenCode

OpenAI-compatible via AI SDK

baseURL in the provider block of opencode.json

Capability lookup through models.dev misses on custom aliases, image input is dropped silently

Copilot in the IDE

Copilot API, BYOK separate

Enterprise BYOK server-side against an OpenAI-compatible endpoint

Code completions stay on Copilot models permanently, BYOK covers chat and agent only

wire_api = "chat" was removed from Codex in February 2026. Every 2025 guide showing you a Chat Completions provider is dead. If your backend has no Responses API you need a bridge, and LiteLLM has use_chat_completions_api for exactly that.

And Claude Code reads ANTHROPIC_BASE_URL from project settings. CVE-2026-21852 describes the case where a hostile repository sets that value and Claude Code fires requests before the trust prompt appears. Anyone rolling out a gateway rolls out managed settings alongside it. Otherwise the redirect you built is available to other people too.

And the gateway Anthropic built itself

Since 29 June 2026 there is the Claude apps gateway, a stateless container running from the same claude binary, backed by Postgres. It signs developers in over OIDC against Entra ID, distributes managed settings centrally, applies spend caps daily, weekly and monthly at organisation, group or individual level, answers 429 on breach, and ships OTLP telemetry to a collector you operate. Microsoft Foundry is a documented upstream alongside Bedrock, Google Cloud and the Anthropic API, with failover between them.

If Claude Code is your primary tool, that is the answer rather than LiteLLM. No translation layer, no model aliasing, no argument about beta headers. The catch: it only speaks Claude, and OpenAI traffic is invisible to it.

My recommendation

I would not put a single gateway out to tender. I would staff three control points.

Application traffic on Azure through the AI gateway in API Management, wired up via Foundry. Mind the preview status and the fact that you build the currency conversion yourself.

Claude Code through the Claude apps gateway with Foundry upstream, once the data zone question is settled. It gives you exactly the per-person spend caps you otherwise lack.

Anything that mixes formats, so mixed coding agents, evaluations, non-Azure models, through LiteLLM. With Postgres, a pinned version, a verified signature and your own egress filter.

For seat traffic you build no gateway at all. You move the controls into the admin consoles: per-user spend limits in Claude, credits and RBAC spend controls in the ChatGPT workspace, budgets in Copilot billing. This is not a technical measure. It is the most important one.

And measure the right number. Anthropic reports around 13 dollars per developer per active day for Claude Code, 150 to 250 a month, with ninety per cent staying under 30 per active day. Cost per developer per active day is the figure you steer with. The monthly total only tells you it is already too late.

What I cannot judge yet

Whether the unified model API in APIM will ever serve /v1/messages. As long as it does not, a second component stays in the picture for Claude Code, and the consolidation you hoped a gateway would deliver happens by halves.

Whether running two gateways costs less in operations than the damage from split reporting. My guess is yes, and I have not yet measured it over a full quarter.

Sources

Microsoft Foundry · Development · Governance & Compliance