Skip to main content

Featured

GPU Memory Sharding with FSDP

GPU Memory Sharding with FSDP

DISTRIBUTED DEEP LEARNING — PART 1/5 Day 1: Demystifying GPU Memory & The ZeRO Revolution — DeepSpeed, FSDP & State Sharding 25 min read Series: The Dharma of Development Distributed DL (Day 1 / 5) Level: Principal / Systems AI Engineer 💥 Context: You attempt to fine-tune or pretrain a 13-billion parameter dense transformer on an 80GB NVIDIA H100 or A100 GPU using standard PyTorch. You set your batch size to 1. You press enter. Within three seconds, your terminal explodes with the most dreaded error in artificial intelligence: torch.cuda.OutOfMemoryError: CUDA out of memory . How is this possible? In 16-bit precision, 13 billion parameters occupy only ~26 GB of disk space. Why can't an 80 GB state-of-the-art GPU train a 26 GB model? Because the naive mental model of machine learning memory is deeply flawed. Today, we demystify the true $16\Phi$ memory footprint of neural networks and master the architecture that made modern large language models possi...

The zen coding

 

The Dharma of Development: Finding Focus in a World of Code

In the fast-paced world of software engineering, we often find ourselves caught between tight deadlines and complex bug reports. Today, we’re stepping back to look at the "Dharma" (the duty) behind our craft—merging ancient wisdom with modern logic.

​📿 The Gita Wisdom

"Karmanye vadhikaraste ma phaleshu kadachana..."

(Bhagavad Gita — Chapter 2, Verse 47)


The Essence:

You have a right to your actions, but you are not entitled to the fruits of those actions.

​💻 The Programmer's Take

​As developers, we often obsess over the end result—the app launch, the green checkmark on a test suite, or the praise from a senior lead. However, the Gita teaches us that our true purpose lies in the execution, not the outcome.

​When you stop worrying about the looming deadline for a moment and focus purely on writing the most elegant, efficient logic possible, the "fruit" (the high-performance application) follows naturally. True mastery comes when we find joy in the process of solving the problem, rather than just hitting the "Deploy" button.

​🐍 The Programming Tip: The Art of Readability

​Since we are building a "Zen" mindset, let’s talk about the foundation of professional code: Readability.

"Code is read much more often than it is written."Guido van Rossum (Creator of Python)





​✨ Today’s Motivation

Don't be afraid of the bugs. A bug is not a failure; it is simply a lesson you haven't learned yet. Stay focused on the process of debugging. When you approach a stack trace with curiosity instead of frustration, the solution reveals itself much faster.

​In Python, writing clean code is a way of being "detached" from your own ego. You aren't writing for yourself; you're writing for the next person who has to maintain your work.

  • Bad Practice: x = 10 (Vague and confusing)
  • The Pro Way: retry_limit = 10 (Clear and intentional)

​By using descriptive variable names, you reduce the "mental load" for others, creating a more harmonious development environment.






Comments