Enterprise AI Adoption: 7 Layers You MUST Consider

Every enterprise AI conversation I'm in eventually reaches the same moment: a stakeholder asks "so which model should we use?" — as if that's the decision. It almost never is. Models are increasingly interchangeable; what separates the enterprises that scale AI from the ones drowning in shadow deployments is the platform wrapped around the model.

That platform has seven layers. Here's the stack I walk clients through, bottom to top. Skipping a layer doesn't make it go away; it makes it a surprise.

Layer 1: The gateway

One OpenAI-compatible endpoint that every application in the company calls. Behind it: multi-provider routing, failover, retries, circuit breakers, and load balancing. When a provider has an outage — and every provider has outages — the gateway reroutes and your apps never notice. When a better model ships next quarter, you change a routing rule, not forty codebases.

The non-negotiable rule this layer enforces: no application ever holds a raw provider API key. Every downstream layer — identity, budgets, observability, guardrails — depends on all traffic flowing through this one door. Build the door first.

Layer 2: Identity and access

Virtual keys per team and per app, mapped to your identity provider — Entra or Okta via SSO and SCIM. Each virtual key carries its own permissions, rate limits, budget, and model allowlist, and it deactivates when the owner offboards, because it's tied to your IdP rather than to a sticky note.

This is also the layer that kills shadow AI, and the mechanism is blunt: revoke the raw keys at the provider level and force everything through the gateway. You can't govern traffic you can't see, and you can't see traffic that goes around you. Make the sanctioned path the only path — and make it the easiest one, or people will expense their own subscriptions again.

Layer 3: The cost control plane

Token spend is the new cloud bill, and it needs the same FinOps machinery: hierarchical budgets from org to business unit to team to app, token metering on every request, and chargeback or showback so consumption has an owner. An unattributed AI bill gets cut at budget season; an attributed one gets defended by the team whose numbers it improves.

The other half of this layer is model tiering policy. Classification and extraction jobs don't need a frontier model — route them to a Haiku-class model at a fraction of the cost, and reserve frontier reasoning for the work that earns it. Enforced centrally at the gateway, tiering is one policy; left to individual teams, it's a hundred inconsistent decisions defaulting to the most expensive option.

Layer 4: Observability

Full request and response logging, latency and error tracing, token counts per call — with PII-aware retention, because prompts contain customer data and your logs inherit every compliance obligation that data carries. When someone asks "why did the assistant say that to a customer," replaying the exact request is the difference between an answer and a shrug.

Export it to the stack you already run — Datadog, Splunk, wherever your on-call engineers already look — rather than standing up an AI-shaped silo. LLM traffic is production traffic; it should page the same people through the same dashboards.

Layer 5: Guardrails and policy

Policy enforcement on both directions of every request: real-time content moderation, PII redaction before prompts leave your boundary, prompt-injection screening on anything untrusted that flows into a model, and hooks into your existing DLP tooling. Sitting at the gateway, these controls apply to every app uniformly — including the ones built after the policy was written.

Prompt injection deserves particular respect the moment a model can both read untrusted content and take actions. If an agent reads emails and can also send them, you've built an attack surface, and it needs the same threat modeling as any other one.

Layer 6: The eval harness

A golden dataset per use case — real, representative inputs with known-good outputs — and regression gates in CI so no model swap or prompt change ships silently. "We upgraded the model and it seems fine" is not a test plan. Without evals, every change is a bet placed on vibes; with them, the routing flexibility Layer 1 gave you becomes safe to actually use.

Note that this is usually a separate tool from the gateway — Braintrust, Maxim, Arize, LangSmith, and friends — wired into the same CI that gates the rest of your code. The gateway sees traffic; the eval harness judges quality. You need both views.

Layer 7: The MCP / tool gateway

The newest layer, and increasingly table stakes: agents don't just generate text, they call tools — and those tool calls need the same centralization the model calls got. An MCP gateway gives you one place for tool routing, OAuth handling, and access control across agent workflows, instead of every agent holding its own credentials to your order management system.

The traffic is coming whether the governance is ready or not — Gartner forecasts that 40% of enterprise applications will embed task-specific agents by the end of 2026. An agent is an identity that acts. If Layer 2 answers "who can call which model," this layer answers the more consequential question: "who can do what."

The uncomfortable summary

Notice what's absent from this stack: the model. Six of the seven layers are gateway plumbing, identity, FinOps, observability, policy, and testing — platform engineering disciplines enterprises already understand, applied to a new kind of traffic. The teams treating AI adoption as a platform build are quietly scaling it; the teams treating it as a procurement decision are accumulating shadow deployments and unattributed invoices.

Model choice is a routing rule. The seven layers around it are the architecture.