Post training - Reinforcement Learning

Mon, 01 Jan 0001 00:00:00 +0000

The post on post-training SFT covered four finetuning experiments on direct and reasoning tasks using LoRA and full-finetuning approaches. This post covers different reinforcement learning (RL) approaches used in post-training.

SFT teaches a model by giving it examples to imitate. The training data contains target responses, and the model learns to generate tokens from the target distribution.

RL takes a different approach. Instead of showing the model exactly how to solve a problem, we let it generate solutions, evaluate how good those solutions are, and update the model so that successful behaviour becomes more likely.

Post training - Supervised Finetuning

Mon, 01 Jan 0001 00:00:00 +0000

In previous posts, I explored how to pretrain a language model. The setup for pretraining used Llama 3.2 1B base model trained on tokyotech-llm/swallow-code-v2 dataset for Fill-in-Middle (FIM) task.

The mistake here was using narrow code-only corpus as pretraining dataset. A more practical pipeline would be to begin with a broadly pretrained base model, optionally continue pretraining it on high-quality code, and then apply SFT using verified prompt–response demonstrations. Pretraining task are used to learn language representation across various tasks.

Post-Training on Blog

Post training - Reinforcement Learning

Post training - Supervised Finetuning