Ultrascale Playbook - Expert Parallelism
Notes on training LLMs using expert parallelism
Llm-Training
Ultrascale Playbook - Context Parallelism
Notes on training LLMs using context parallelism
Ultrascale Playbook - Tensor and Sequence Parallelism
Notes on training LLMs using tensor and sequence parallelism
Ultrascale Playbook - Pipeline Parallelism
Notes on training LLMs using pipeline parallelism
Pydata MCR talk on training LLMs
My talk on training LLMs at Pydata MCR