Projects

One of the projects I am looking forward to is training an LLM from scratch. For this project, I am thinking about creating a mini LLM version of StarCoder2 model. StarCoder2 is a CodeLLM. To limit the scope here’s a brief outline of the project Mini LLM, maybe 200M parameter model as a start Focusing only on Python as a language The Chinchilla paper suggests using roughly 20x the number of parameters in tokens for optimal training. For my 200M parameter model, this means ~4B tokens would be ideal. To start, I’ll experiment with a smaller slice of 500M tokens. ...

I am using Zed editor especially the Vim mode for quite some time. VSCode editor my previous thing. The experience on VSCode was laggy and slow using VSCodeVim. I’ll blame my machine for that, rather than my not-so-great Vim skills. I quite like the Camouflage VSCode extension by zeybek. This extension hides the sensitive environment variables in the .env file by hiding their values. No more “oops” moment. Perfect excuse to learn Rust - Create a similar extension for Zed. ...