Ultrascale Playbook - Pipeline Parallelism
Notes on training LLMs using pipeline parallelism
Llm-Training
Pydata MCR talk on training LLMs
My talk on training LLMs at Pydata MCR
Distributed communication for GPUs (part 2)
Introduction to collective communication operations used for distributed training.
Distributed communication for GPUs (part 1)
Introduction to distributed communication for GPUs.
Choosing a batch size and provider for LLM training
Notes on choosing appropriate batch size and compute for training LLMs