Llm

Implement ZeRO sharding strategies using PyTorch

Implement various data parallelism strategies using PyTorch

Notes on training LLMs using expert parallelism

Notes on training LLMs using context parallelism

Notes on training LLMs using tensor and sequence parallelism