Pydata MCR talk on training LLMs
My talk on training LLMs at Pydata MCR
Llm-Training
Distributed communication for GPUs (part 2)
Introduction to collective communication operations used for distributed training.
Distributed communication for GPUs (part 1)
Introduction to distributed communication for GPUs.
Choosing a batch size and provider for LLM training
Notes on choosing appropriate batch size and compute for training LLMs
Ultra-scale Playbook - ZeRO Sharding
Notes on training LLMs using sharding strategies