Determining VRAM Requirements for Local LLM Deployment
When deploying a local LLM, the primary consideration is whether it can reside entirely within your GPU's memory. The outcome hinges on the model's size, its quantization level, and the intended context length. This guide provides a practical framework for selecting the appropriate amount of VRAM.
The Impact of Quantization on VRAM
Quantization lowers the precision required to store model weights. Reducing the bit-width shrinks the model size and decreases VRAM consumption, albeit with a slight trade-off in output quality.
| Quant | Bits per weight | Typical use |
|---|---|---|
| Q8_0 | 8 | Very high quality |
| Q6_K | ~6.6 | Very good quality |
| Q5_K_M | ~5.5 | Good quality and size |
| Q4_K_M | ~4.5 | Good balance of size and quality |
| Q3_K_M | ~3.5 | Lower VRAM, more quality loss |
Q4_K_M is frequently selected when VRAM resources are constrained. Conversely, if more VRAM is available, opting for Q5 or Q6 allows for running the same model with reduced quantization pressure.
Estimated VRAM Needs by Model Size
The following figures are rough approximations for model weights alone. The total VRAM required will be higher, as the runtime environment, KV cache, and context window also consume memory.
| Model size | Q8_0 | Q6_K | Q4_K_M | Q3_K_M |
|---|---|---|---|---|
| 4B | ~5 GB | ~4 GB | ~3 GB | ~2.5 GB |
| 8B | ~9 GB | ~7 GB | ~5.5 GB | ~4.5 GB |
| 12B | ~13 GB | ~10 GB | ~8 GB | ~6.5 GB |
| 14B | ~16 GB | ~12 GB | ~9 GB | ~7.5 GB |
| 27B | ~30 GB | ~22 GB | ~17 GB | ~13 GB |
| 32B | ~36 GB | ~27 GB | ~20 GB | ~16 GB |
| 70B | ~80 GB | ~60 GB | ~42 GB | ~34 GB |
These values are estimates rather than absolute limits. Variations in model architecture and quantization formats can influence the actual memory footprint.
Capabilities Across Different VRAM Tiers
| VRAM | Practical range | Current examples |
|---|---|---|
| 8 GB | Small models around 4B to 9B | Gemma 4 E4B, Qwen3.5 9B |
| 12 GB | Small to mid-sized models around 9B to 14B | Gemma 4 12B, Qwen3.5 9B |
| 16 GB | 12B to 27B with lower quantization | Gemma 4 26B-A4B, Qwen3.6 27B at Q4 |
| 24 GB | 27B to 35B at Q4 to Q6 | Qwen3.8 27B, Gemma 4 31B |
| 32 GB | 27B to 35B at higher quantization | Qwen3.8 27B, Gemma 4 31B |
| 48 GB | Large dense models at lower quantization | 70B-class models at Q3 to Q4 |
| 80 GB | Large dense models at higher quantization | 70B-class models at Q4 to Q6 |
These ranges apply to models where the weights fit within the GPU. Large MoE models operate differently: while only a subset of parameters is active per token, the entire weight set must still be stored. Consequently, a model with 100B or more total parameters will not fit into a VRAM budget matched to its active parameters alone.
MoE Models
Mixture-of-Experts models consist of multiple parameter groups known as experts. Since only specific experts are engaged for each token, inference can be more efficient than with a dense model of the same total parameter count.
However, inactive experts remain part of the model structure. As a result, large MoE models may demand significantly more memory than their active parameter count implies. Extremely large models might necessitate multiple GPUs or offloading to system RAM.
Context Length and VRAM Usage
Model weights constitute only one part of the memory equation. The KV cache expands as context length increases; thus, running the same model with a 64K context requires substantially more VRAM than with a 4K context.
- Extended context windows demand more VRAM.
- The precision of the KV cache influences memory consumption.
- Batch size and concurrent users further increase memory usage.
- Reserve some VRAM for the runtime rather than filling the GPU entirely with model weights.
Practical Recommendations
- Verify the actual size of the specific quantized model you intend to run.
- Do not equate the model file size with the exact VRAM requirement. Allow headroom for the KV cache and runtime.
- If a model exceeds available VRAM, part of it can be offloaded to system RAM, though this typically slows down inference.
- For long-context or agentic workloads, allocate more VRAM than the model weights alone require.
- Multiple GPUs can distribute a model's load if a single GPU lacks sufficient VRAM.
Deploy on DaDesktop
There is no need to purchase hardware to run a local LLM. DaDesktop provides a cloud desktop equipped with the required VRAM, allowing you to execute the model directly without owning the underlying hardware.
Select a VRAM tier suitable for your model, load it, and begin using it. This eliminates the need for setup, hardware purchases, or driver troubleshooting. Visit available GPUs to view your options.