Llm

The post on post-training SFT covered four finetuning experiments on direct and reasoning tasks using LoRA and full-finetuning approaches. This post covers different reinforcement learning (RL) approaches used in post-training. SFT teaches a model by giving it examples to imitate. The training data contains target responses, and the model learns to generate tokens from the target distribution. RL takes a different approach. Instead of showing the model exactly how to solve a problem, we let it generate solutions, evaluate how good those solutions are, and update the model so that successful behaviour becomes more likely. ...

In previous posts, I explored how to pretrain a language model. The setup for pretraining used Llama 3.2 1B base model trained on tokyotech-llm/swallow-code-v2 dataset for Fill-in-Middle (FIM) task. The mistake here was using narrow code-only corpus as pretraining dataset. A more practical pipeline would be to begin with a broadly pretrained base model, optionally continue pretraining it on high-quality code, and then apply SFT using verified prompt–response demonstrations. Pretraining task are used to learn language representation across various tasks. ...

One of the projects I am looking forward to training an LLM from scratch. For this project, I am creating a mini LLM version of StarCoder2 model. StarCoder2 is a CodeLLM. To limit the scope here’s a brief outline of the project Mini LLM, maybe 200M parameter model as a start Focusing only on Python as a language The Chinchilla paper suggests using roughly 20x the number of parameters in tokens for optimal training. For my 200M parameter model, this means ~4B tokens would be ideal. To start, I’ll experiment with a smaller slice of 500M tokens. ...

It’s time to get the wallets out 💸💸💸💸 Useful Links Code Github: https://github.com/dudeperf3ct/minicode-llm/tree/main/codellm_pretrain/torch_titan Experiment W&B for project: https://wandb.ai/dudeperf3ct/torchtitan ...

Now that there is a pretrained dataset containing Python source code in form of text, next task would be to create a tokenizer specific to the code. Tokenization Tokenization is the process of converting text into a numerical representation that a model can process. The simplest possible encoding is mapping each character to its ASCII value: >>> list("hello world".encode('ascii')) [104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100] ASCII works, but it is limited to 128 symbols. Modern text includes code comments, Unicode identifiers, and emojis. That’s where Unicode comes in. ...