https://dudeperf3ct.github.io/tags/sft/ Recent content in Sft on Blog Hugo en dudeperf3ct https://dudeperf3ct.github.io/projects/post_training_llm_sft/ Mon, 01 Jan 0001 00:00:00 +0000 https://dudeperf3ct.github.io/projects/post_training_llm_sft/ <p>In previous posts, I explored how to pretrain a language model. The setup for pretraining used Llama 3.2 1B base model trained on <a href="https://huggingface.co/datasets/tokyotech-llm/swallow-code-v2"><code>tokyotech-llm/swallow-code-v2</code></a> dataset for Fill-in-Middle (FIM) task.</p> <p>The mistake here was using narrow code-only corpus as pretraining dataset. A more practical pipeline would be to begin with a broadly pretrained base model, optionally continue pretraining it on high-quality code, and then apply SFT using verified prompt–response demonstrations. Pretraining task are used to learn language representation across various tasks.</p>