The Pre-training Recipe
3.3B words, 1M steps, warmup plus linear decay, and 128 before 512
Sign in to keep going
Create a free account or sign in to continue. Your progress stays saved on this device.
3.3B words, 1M steps, warmup plus linear decay, and 128 before 512
Create a free account or sign in to continue. Your progress stays saved on this device.