GitHub · GPT · Nvidia · Hugging Face
The GIF below indicates the difference between the dense and fused-chunked approaches
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
Hugging Face has open-sourced the chunked-loss implementation: github.com/CompactifAI/Full-Chunked-KL-Loss.
Key facts
- At a sequence length of 32K and batch size 4, the teacher-probability tensor alone has shape 4 × 201,088 × 32,768; in bfloat16, that's already about 50GB of VRAM for a single tensor
- At 256K tokens, the fully chunked loss uses 11.6 GiB against 134.2 GiB for the next-best chunked variant, and is about 3.3× faster per iteration at that length
- Distilling a GPT-OSS 20B model at a 32,768-token context, the memory freed by the fused loss let the setup shrink from four GPU nodes down to one
- Deploying these large models is expensive: the recent Kimi-K3 model has 2.8 trillion parameters and needs roughly 3TB of VRAM to load
Summary
Knowledge distillation, training a smaller student model to match the performance of a larger teacher, is a well-known technique in Machine Learning. The distillation step is what decides most of the final quality, but it's also usually the most expensive part of the pipeline. The standard setup, online distillation using the Kullback-Leibler divergence loss (KL loss), keeps both the teacher and the student loaded at the same time. As a practical example, gpt-oss-120b has a vocabulary of 201,088 tokens. Dense KL spikes to roughly 250GB, above a single H200's 141GB capacity.