How GPU Inference Works
During model inference, a GPU stores model state and temporary execution data in high-bandwidth device memory, then runs parallel tensor and matrix operations that power prompt processing and token generation.
Explore the interactive architecture →Device memory
High-bandwidth memory holds model weights, KV cache, activations and temporary buffers. Available memory capacity constrains model size, context length and the number of sequences that can be processed together.
Parallel compute
GPU compute engines execute the tensor and matrix operations used by the model. Prefill and decode stress hardware differently, but both depend on coordinated movement of data between device memory and compute units.
Performance bottlenecks
Inference performance is limited by a combination of compute throughput, memory bandwidth and memory capacity. The active bottleneck changes with model shape, prompt length, batch size, runtime optimizations and hardware characteristics.