TP/PP with Different VRAM Cards

[sieveLau]

I have a 4070 Ti Super (16G) and I'm considering buying another card. I wonder if I buy a Tesla L4 (24G), what will happen using TP and PP?

Based on Googling, is that I can only use 32G in TP while full 40G in PP?

Thanks for any suggestions

[Shyboy0499]

Short version of what happens with a 4070 Ti Super (16 GB) + Tesla L4 (24 GB):

TP (tensor parallel): shards every layer evenly across ranks → limited by the smallest GPU. You only get ~16 GB usable per rank (~32 GB total), and the L4's extra 8 GB is wasted. TP also wants fast interconnect, and these are two PCIe cards with no NVLink between them — TP over PCIe is slow for LLMs.

PP (pipeline parallel): splits the model by layers, each rank holds only its slice → both cards' memory adds up (16 + 24 = 40 GB usable). But throughput is bounded by the slowest stage — and the L4 is a much weaker card than the 4070 Ti Super, so the pipeline runs at the L4's pace. PP doesn't all-reduce every layer like TP, so PCIe is fine.

So your "32 G in TP, 40 G in PP" intuition is right in spirit: TP wastes to the smaller card; PP sums memory but slows to the weaker card.

Practical guidance:

• If you need capacity (a bigger model) → PP gets you the full 40 GB, at the L4's speed. Fits more, generates slower.
• If you need throughput → don't mix. Run the model on the 4070 Ti Super alone (16 GB fits e.g. a ~13B at Q4) and use the L4 as a second, independent instance/batch. Two separate deployments beat one heterogeneous TP/PP pair almost every time.
• vLLM does have newer experimental support for uneven sharding (--rank-tp-ratio), but for two cards this different in compute (not just VRAM), splitting into independent instances is the better call.

One more caveat: if you go PP, also set the pipeline_parallel_size to 2 and give each worker one card (CUDA_VISIBLE_DEVICES per rank), and check stage imbalance — the memory benefit is real, the speed hit is too.