Ultrascale Playbook - Tensor and Sequence Parallelism
Notes on training LLMs using tensor and sequence parallelism
Llm
Ultrascale Playbook - Pipeline Parallelism
Notes on training LLMs using pipeline parallelism
Pydata MCR talk on training LLMs
My talk on training LLMs at Pydata MCR
Choosing a batch size and provider for LLM training
Notes on choosing appropriate batch size and compute for training LLMs
Ultra-scale Playbook - ZeRO Sharding
Notes on training LLMs using sharding strategies