Skip to main content

Featured

GPU Memory Sharding with FSDP

GPU Memory Sharding with FSDP

DISTRIBUTED DEEP LEARNING — PART 1/5 Day 1: Demystifying GPU Memory & The ZeRO Revolution — DeepSpeed, FSDP & State Sharding 25 min read Series: The Dharma of Development Distributed DL (Day 1 / 5) Level: Principal / Systems AI Engineer 💥 Context: You attempt to fine-tune or pretrain a 13-billion parameter dense transformer on an 80GB NVIDIA H100 or A100 GPU using standard PyTorch. You set your batch size to 1. You press enter. Within three seconds, your terminal explodes with the most dreaded error in artificial intelligence: torch.cuda.OutOfMemoryError: CUDA out of memory . How is this possible? In 16-bit precision, 13 billion parameters occupy only ~26 GB of disk space. Why can't an 80 GB state-of-the-art GPU train a 26 GB model? Because the naive mental model of machine learning memory is deeply flawed. Today, we demystify the true $16\Phi$ memory footprint of neural networks and master the architecture that made modern large language models possi...
Post a Comment
Read more

The Architect's Intent

System core about the dev

"Yoga is skill in action." — Bhagavad Gita 2.50

Om: The Origin


I am a developer who believes that engineering is more than just shipping features; it is a discipline of clarity, logic, and structure. My daily battlefield involves architecting resilient backend systems, diving deep into CPython internals, and extracting pure signal from the noise.

Whether I am forging a massive 3,000-line Selenium automation engine or testing the performance matrices of Python against Rust for data parsing, my goal remains the same: writing code that stands the test of time.

The Infinite Game

The Dharma of Development is the live documentation of my ongoing pursuit of mastery. This is the architecture of success. Here, we do not just memorize syntax. We explore the "Why" behind the machinery.

We look at memory allocation, mathematical computing, data science fundamentals, and the architectural decisions that separate a standard script from a legacy system.

The Arsenal

My technical focus revolves around high-performance execution, deep algorithmic logic, and automation.

Python (CPython) System Architecture Data Science & Math PostgreSQL / pgAdmin Selenium

Comments

Post a Comment

?: "90px"' frameborder='0' id='comment-editor' name='comment-editor' src='' width='100%'/>

Popular Posts