Ultrascale Playbook - Context Parallelism
Notes on training LLMs using context parallelism
Llm-Training
Ultrascale Playbook - Tensor and Sequence Parallelism
Notes on training LLMs using tensor and sequence parallelism
Ultrascale Playbook - Pipeline Parallelism
Notes on training LLMs using pipeline parallelism
Pydata MCR talk on training LLMs
My talk on training LLMs at Pydata MCR
Distributed communication for GPUs (part 2)
Introduction to collective communication operations used for distributed training.