AI Gateway Architecture on Kubernetes
An AI gateway is the traffic entry point for model requests. Kubernetes Gateway API objects configure matching and backend selection, while the runtime gateway or proxy executes those rules and delegates specialized policy checks to external processors.
Explore the interactive architecture →Configuration is not a network hop
Gateway and HTTPRoute are configuration objects. They describe listeners, matching rules and backend references. A gateway controller reconciles that configuration into the runtime proxy, so HTTPRoute should not be modeled as a request-time processing stage.
Traffic and external processing
The gateway accepts a request, matches a route and selects a backend. Authentication, shared rate limits and guardrails can run as external processing callouts. These services support request processing but remain separate runtime dependencies.
- Identity validates caller context and claims.
- Rate limits enforce shared request or token budgets.
- Guardrails apply configured input or output safety checks.
Service and InferencePool backends
A Service is a standard Kubernetes backend. An InferencePool is an inference-aware backend abstraction. When a route targets an InferencePool, the gateway can consult an inference router or Endpoint Picker to select a suitable serving endpoint, then the gateway forwards the request to that endpoint.