Llama · hugovergnes.github.io
Tuning a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
Somewhere between “nanoGPT toy” and “you need a research lab” there’s a large, under-described region where one person with a few thousand dollars can train a meaningful model.
Key facts
- The result is a 3.8B-parameter model scoring 0.384 on CORE, trained on 65B tokens in 43 hours for $998
- Dropping the gate projection (SwiGLU → relu², two matmuls instead of three) on the small model: 183,035 → 214,173 tok/s and 6 GB less VRAM
- Keeping the optimizer master weights in bf16 rather than fp32 cut VRAM 27% and raised throughput from 640K to 1.4M tok/s on the 1.5B config
- The quality cost is real but small: CORE 0.22 vs 0.23 at 4,000 steps
Summary
The reporter wanted to see language and understanding emerge from random weights for themselves, and to learn the parts you can only learn by starting from scratch. The result is a 3.8B-parameter model scoring 0.384 on CORE, trained on 65B tokens in 43 hours for $998. What follows is what worked, what didn’t, and what the reporter still don’t know. Their model is larger than nanochat d32 and took similar wall-clock time. But for roughly the same money as nanochat’s $1,000 configuration, this lands meaningfully ahead of it.