Enterprise AI governance: the four questions a CIO asks
Every executive review of AI spend comes down to four questions. Each answer here points at a shipped mechanism you can verify — not a roadmap item.
Granting model access is easy; governing it is not. The four questions below decide whether an AI programme survives its first audit: who is using which model, what it costs and who pays, whether data left the network, and what a provider change costs in engineering time.
Who is using which model, and how much?
Every request carries a key, a key group and a model. GateLLM records input, output and cache-hit tokens per request at one-minute granularity, and exposes daily and monthly series per key, group and model. Admission control decides which clients may connect at all, by request header.
- •Usage and cost break down by person, key, key group and model — no log reconstruction needed.
- •Client admission control gates by User-Agent across global, group and key scopes.
- •Request-level records keep the identifier needed to trace one call end to end.
What does it cost, and who pays for it?
Spend is metered against a rate card you own, not a vendor list price. Pin your negotiated prices, tag calls for chargeback, export per-window reports, and set daily or monthly cost quotas that reject requests when a budget is exhausted. GateLLM adds no token markup on top of provider prices.
- •Time-windowed rate-card snapshots keep an old report reproducible.
- •Tag-based chargeback turns AI spend into an allocable ledger.
- •Daily and monthly quotas return 429 when the cap is hit, so a runaway agent stops at a number you chose.
Did data leave the network, and can we prove it?
You run the gateway, so prompt and response traffic never passes through us. Under BYOK, provider keys live in gateway memory only. Credentials and PII are detected on the request path and can be logged, masked or blocked per scope, and stored log bodies are redacted at every egress point.
- •Requests go only to the model endpoints you configure — including self-hosted vLLM or Ollama upstreams.
- •Request logs are off by default; revealing raw content is admin-only and audited.
- •SSO via OIDC or SAML plus SCIM provisioning; per-tenant keys and quotas.
If a provider fails or we change models, how much code changes?
A provider 5xx or timeout is retried on another provider without touching application code. Four chat-protocol families translate in both directions, so a client keeps its own SDK. Model identity is mapped at the gateway — by access key, by header or through a catch-all rule — and capability metadata comes from a registry.
- •Cross-provider fallback on 5xx or timeout, with no client change.
- •Ingress allowlists and catch-all mappings keep one model name stable while the backend changes.
- •Capability metadata loads from a registry, so model drift does not require a release.