https://muthukamalan.github.io/ • This is a web feed, also known as an Atom feed. Subscribe by copying the URL from the address bar into your newsreader. Visit About Feeds to learn more and get started. It's free. Visit website Recent posts Updated on $DATE one-dark-pro date Vllm Friendly Neighbour Programmer Zola2026-07-21T01:01:35+00:00https://muthukamalan.github.io/tags/vllm/atom.xml 2026-07-21T01:01:35+00:00 2026-07-21T01:01:35+00:00 Muthukamalan https://muthukamalan.github.io/blog/llm-inference/ <details> <summary>Table Of Content</summary> <!-- toc --> </details> <h1 id="1-overview"><a class="header-anchor no-hover-padding" href="#1-overview" aria-label="Anchor link for: 1-overview"><span class="link-icon" aria-hidden="true"></span></a> 1. Overview</h1> <p>Large LLMs like Llama-3-70b or Falcon 180B may not fit in a single GPU.</p> <p>If training/serving a model on a single GPU is too slow or if the model’s weights do not fit in a single GPU’s memory, transitioning to a multi-GPU setup may be a viable option.</p> <p>But serving large language models (LLMs) with multiple GPUs in a distributed environment might be a challenging task.</p> <h1 id="2-checking-the-model-footprint-of-the-model"><a class="header-anchor no-hover-padding" href="#2-checking-the-model-footprint-of-the-model" aria-label="Anchor link for: 2-checking-the-model-footprint-of-the-model"><span class="link-icon" aria-hidden="true"></span></a> 2. Checking the Model Footprint of the Model</h1> <p>Before deploying a model in a distributed environment, it is important to check the memory footprint of the model.</p> <p>To begin estimating how much vRAM is required to serve your LLM, we can use these tools:</p> <ul> <li><a class="external" rel="external" href="https://huggingface.co/spaces/hf-accelerate/model-memory-usage">HF Model Memory Usage</a></li> <li><a class="external" rel="external" href="https://rahulschand.github.io/gpu_poor/">GPU Poor vRAM Calculator</a></li> <li><a class="external" rel="external" href="https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator">LLM Model VRAM Calculator</a> only for quantization models</li> <li><a class="external" rel="external" href="https://llm.extractum.io/">LLM Explorer</a> to check raw model vRAM size consumption</li> </ul> <!-- https://github.com/rh-aiservices-bu/gpu-partitioning-guide#14-pros-and-cons-of-gpu-partitioning-methods --> <!-- https://github.com/rh-aiservices-bu/multi-gpu-llms#1-overview --> <h1 id="3-gpu-partitioning-overview"><a class="header-anchor no-hover-padding" href="#3-gpu-partitioning-overview" aria-label="Anchor link for: 3-gpu-partitioning-overview"><span class="link-icon" aria-hidden="true"></span></a> 3. GPU Partitioning Overview</h1> <p><img src="https://github.com/rh-aiservices-bu/gpu-partitioning-guide/raw/main/assets/gpu-sharing-overview.png" alt="Overview" /></p> <h2 id="time-slicing"><a class="header-anchor no-hover-padding" href="#time-slicing" aria-label="Anchor link for: time-slicing"><span class="link-icon" aria-hidden="true"></span></a> Time-slicing</h2> <h2 id="mig"><a class="header-anchor no-hover-padding" href="#mig" aria-label="Anchor link for: mig"><span class="link-icon" aria-hidden="true"></span></a> MIG</h2> <h2 id="mps"><a class="header-anchor no-hover-padding" href="#mps" aria-label="Anchor link for: mps"><span class="link-icon" aria-hidden="true"></span></a> MPS</h2> <h1 id="pros-and-cons-of-gpu-partitioning-methods"><a class="header-anchor no-hover-padding" href="#pros-and-cons-of-gpu-partitioning-methods" aria-label="Anchor link for: pros-and-cons-of-gpu-partitioning-methods"><span class="link-icon" aria-hidden="true"></span></a> Pros and Cons of GPU Partitioning Methods</h1> <h1 id="requirements-gpu-nodes-with-gpus"><a class="header-anchor no-hover-padding" href="#requirements-gpu-nodes-with-gpus" aria-label="Anchor link for: requirements-gpu-nodes-with-gpus"><span class="link-icon" aria-hidden="true"></span></a> Requirements: GPU Nodes with GPUs</h1> <h1 id="validate-and-gpu-partioning"><a class="header-anchor no-hover-padding" href="#validate-and-gpu-partioning" aria-label="Anchor link for: validate-and-gpu-partioning"><span class="link-icon" aria-hidden="true"></span></a> Validate and GPU Partioning</h1> <h1 id="testing-with-llms"><a class="header-anchor no-hover-padding" href="#testing-with-llms" aria-label="Anchor link for: testing-with-llms"><span class="link-icon" aria-hidden="true"></span></a> Testing with LLMs</h1> <h1 id="nvidia-gpu-operator"><a class="header-anchor no-hover-padding" href="#nvidia-gpu-operator" aria-label="Anchor link for: nvidia-gpu-operator"><span class="link-icon" aria-hidden="true"></span></a> Nvidia GPU Operator</h1> <h1 id="links-of-interest"><a class="header-anchor no-hover-padding" href="#links-of-interest" aria-label="Anchor link for: links-of-interest"><span class="link-icon" aria-hidden="true"></span></a> Links of Interest</h1> <ul> <li><a class="external" rel="external" href="https://rahulschand.github.io/gpu_poor/">VRAM Calculator</a></li> <li><a class="external" rel="external" href="https://blog.vllm.ai/2023/06/20/vllm.html"></a></li> <li><a class="external" rel="external" href="https://towardsdatascience.com/choosing-the-right-gpu-for-deep-learning-on-aws-d69c157d8c86"></a></li> </ul>

Let's Serve better with KServe