GPU Memory Sharding with FSDP
DISTRIBUTED DEEP LEARNING — PART 1/5 Day 1: Demystifying GPU Memory & The ZeRO Revolution — DeepSpeed, FSDP & State Sharding 25 min read Series: The Dharma of Development Distributed DL (Day 1 / 5) Level: Principal / Systems AI Engineer 💥 Context: You attempt to fine-tune or pretrain a 13-billion parameter dense transformer on an 80GB NVIDIA H100 or A100 GPU using standard PyTorch. You set your batch size to 1. You press enter. Within three seconds, your terminal explodes with the most dreaded error in artificial intelligence: torch.cuda.OutOfMemoryError: CUDA out of memory . How is this possible? In 16-bit precision, 13 billion parameters occupy only ~26 GB of disk space. Why can't an 80 GB state-of-the-art GPU train a 26 GB model? Because the naive mental model of machine learning memory is deeply flawed. Today, we demystify the true $16\Phi$ memory footprint of neural networks and master the architecture that made modern large language models possi...
