Writing
Notes from training language models, mostly about the things that went wrong and what they taught me.
- Only teach what it can learn
My first multi-task RL run learned almost nothing about math. The fix was in the data, not the algorithm.
- The model that lost its place
Our CUDA-graph fast path silently dropped the positional encoding, and nothing threw an error.
- 243 missing tensors
My fine-tuned model kept repeating the same few words. I spent a while on the training before I found the bug in how the checkpoint was loaded.
- Training where the internet isn't
Lessons from building a training setup for machines that will never see the internet.