GPU LLM Inference
WIP
1. Overview
Large LLMs like Llama-3-70b or Falcon 180B may not fit in a single GPU.
If training/serving a model on a single GPU is too slow or if the model’s weights do not fit in a single GPU’s memory, transitioning to a multi-GPU setup may be a viable option.
But serving large language models (LLMs) with multiple GPUs in a distributed environment might be a challenging task.
2. Checking the Model Footprint of the Model
Before deploying a model in a distributed environment, it is important to check the memory footprint of the model.
To begin estimating how much vRAM is required to serve your LLM, we can use these tools:
- HF Model Memory Usage
- GPU Poor vRAM Calculator
- LLM Model VRAM Calculator only for quantization models
- LLM Explorer to check raw model vRAM size consumption
3. GPU Partitioning Overview
