How to load the model successfully through multi-card in vllm?

[SmallHappyJerry]

How to load the model successfully through multi-card in vllm? How can I learn and understand this problem？Is there a recommended study blog?

[Shyboy0499]

The flag you want is --tensor-parallel-size (CLI) / tensor_parallel_size (Python LLM(...) API). It shards the model across N GPUs on one node.

Working examples

# 2 GPUs
vllm serve meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 2

from vllm import LLM
llm = LLM("meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=2)

Rules of thumb that avoid the common failures

1. TP size must be ≤ the number of visible GPUs. vLLM won't always stop you up front — a too-large TP can hang silently or die with a cryptic CUDA error. Pin the devices explicitly:

   CUDA_VISIBLE_DEVICES=0,1 vllm serve $MODEL --tensor-parallel-size 2

2. Use identical GPUs in one TP group. TP shards every layer evenly across ranks, so the group is only as good as its smallest/slowest GPU — mixing VRAM sizes wastes memory and mixing architectures misbehaves. (The experimental --rank-tp-ratio flag supports uneven sharding, but the default path assumes uniform GPUs.)
3. Keep the TP group on one node with fast interconnect (NVLink/NVSwitch). TP needs all-gather/ring comms every layer; over PCIe or a network it becomes slow.
4. Multi-node means pipeline parallel or Ray, not naive TP across machines. --pipeline-parallel-size splits layers across nodes (far less bandwidth-hungry), and multi-node vLLM runs on Ray.

How to learn this (recommended path)

• vLLM docs — Parallelism Scaling — the current home for "how TP/PP/DP work and which to pick."
• vLLM docs — Distributed Troubleshooting — the exact error patterns people hit (GPU not recognized, hangs).
• For the concepts behind the scenes, a tensor-parallelism explainer (e.g. Hugging Face's blog on tensor parallelism) walks through how weights and activations get sharded.

One more tip: if load time scales with TP size, convert to a sharded checkpoint (save_sharded_state.py / --load-format sharded_state) so each rank reads only its own shard.

Start with parallelism_scaling, run the 2-GPU command above, then read distributed_troubleshooting when something breaks — you'll cover the whole space faster than any single blog post.