1. Overview

Large LLMs like Llama-3-70b or Falcon 180B may not fit in a single GPU.

If training/serving a model on a single GPU is too slow or if the model’s weights do not fit in a single GPU’s memory, transitioning to a multi-GPU setup may be a viable option.

But serving large language models (LLMs) with multiple GPUs in a distributed environment might be a challenging task.

2. Checking the Model Footprint of the Model

Before deploying a model in a distributed environment, it is important to check the memory footprint of the model.

To begin estimating how much vRAM is required to serve your LLM, we can use these tools:

• HF Model Memory Usage
• GPU Poor vRAM Calculator
• LLM Model VRAM Calculator only for quantization models
• LLM Explorer to check raw model vRAM size consumption

3. GPU Partitioning Overview

Time-slicing

MIG

MPS

Pros and Cons of GPU Partitioning Methods

Requirements: GPU Nodes with GPUs

Validate and GPU Partioning

Testing with LLMs

Nvidia GPU Operator

Links of Interest

• VRAM Calculator