ZeRO Sharding Revisited - Implementations using PyTorch
Implement ZeRO sharding strategies using PyTorch
Python in practice - Config driven design
Patterns in python code design
Data Parallelism Revisited - Implementations using PyTorch
Implement various data parallelism strategies using PyTorch
Ultrascale Playbook - Expert Parallelism
Notes on training LLMs using expert parallelism
Ultrascale Playbook - Context Parallelism
Notes on training LLMs using context parallelism