Ultrascale Playbook - Tensor and Sequence Parallelism
Notes on training LLMs using tensor and sequence parallelism
Llm-Training
Ultrascale Playbook - Pipeline Parallelism
Notes on training LLMs using pipeline parallelism
Pydata MCR talk on training LLMs
My talk on training LLMs at Pydata MCR
Distributed communication for GPUs (part 2)
Introduction to collective communication operations used for distributed training.
Distributed communication for GPUs (part 1)
Introduction to distributed communication for GPUs.