Yasaswi Kapila
I work on large language models at Neuralix.AI: model architecture, pretraining, and post-training with fine-tuning and reinforcement learning.
Recently that has included a roughly 10B-parameter mixture-of-experts model, a pretraining stack built from scratch, and, in my own time, multi-task reinforcement learning. I write about what I learn here.
Recent writing
- Only teach what it can learn
My first multi-task RL run learned almost nothing about math. The fix was in the data, not the algorithm.
- The model that lost its place
Our CUDA-graph fast path silently dropped the positional encoding, and nothing threw an error.
- 243 missing tensors
My fine-tuned model kept repeating the same few words. I spent a while on the training before I found the bug in how the checkpoint was loaded.
- Training where the internet isn't
Lessons from building a training setup for machines that will never see the internet.
Off the clock
- Driving
- Long drives in my red Polo.
- Sport
- Tennis, Arsenal, the NBA and F1.
- Reading
- Manga and light novels, plus anime and films.
- Kali
- My dog, back home.