LLM Inference Architecture
A model server executes selected requests efficiently by scheduling work, managing accelerator memory and generating tokens. The runtime coordinates request admission, batching, prefill, autoregressive decode and streaming without turning each optimization into a separate external service.
Explore the interactive architecture →Request scheduling and continuous batching
Incoming requests wait in a queue before the runtime scheduler admits work. Continuous batching lets active sequences join or leave a batch between iterations, improving accelerator utilization while the runtime manages latency and capacity limits.
Prefill and decode
Prefill processes the input prompt and creates the initial key-value state. Decode then generates output autoregressively, one iteration at a time. Generated tokens can stream to the client while decode continues.
KV cache and runtime capabilities
The KV cache stores attention state reused across decode iterations. Prefix caching, chunked prefill and KV-cache management reduce repeated work, control memory pressure and improve throughput. These are capabilities inside the inference runtime rather than external dependencies.