Current Focus: Advanced LLM Finetuning
Goal: Understand the intricacies of PEFT and LoRA to build domain-specific coding models. Resources: Hugging Face Course, Stanford CS224N
I’m currently spending my weekends fine-tuning Llama 3 8B on a custom dataset of Rust and Astro codebases. The goal is to create a model that understands my specific coding style and project preferences.
What I’ve Learned So Far
- Data Quality is King: Spending 80% of time cleaning datasets yields better results than tweaking hyperparameters.
- Quantization: The trade-offs between 4-bit and 8-bit quantization are subtle but impactful for coding tasks.
- Evaluation: Creating a robust evaluation pipeline is harder than the training itself.
Next Up
- CUDA Programming: I want to write my own kernels to understand how the underlying matrix multiplications work.
- Distributed Training: Exploring how to train models across multiple GPUs.
Recently Completed
- System Design Course: Refreshed my knowledge on distributed systems and scalability patterns.
- GraphQL Mastery: Built a complex federation gateway to deeply understand the pros and cons of GraphQL.