Inference-Aware Routing on Kubernetes
Ordinary load balancing assumes requests and replicas are broadly interchangeable. LLM requests vary greatly in compute and memory cost, while replica queues, active sequences and cached prefixes make effective capacity different at request time.
Explore the interactive architecture →Why ordinary load balancing is not enough
A short prompt and a long-context request require different amounts of work. Two model replicas can also have different queue depth, active sequence count, model availability or useful cached prefixes. A static round-robin decision cannot account for that changing serving state.
InferencePool and Endpoint Picker
InferencePool represents a group of serving endpoints as an inference-aware backend. During backend selection, the gateway can call an Endpoint Picker or inference router with request context. The router evaluates eligible endpoints and returns a selected endpoint; the gateway remains responsible for forwarding the user request.
Runtime signals
Endpoint selection can consider readiness, queue state, active load, model or adapter availability and cache locality. The exact signals and scoring policy depend on the implementation, but the architectural role stays the same: make a request-time placement decision from current serving state.