Tech · Hugging Face
Granite 4.1 LLMs: How They’re Assembled
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
TL;DR, Granite 4.1 is a family of dense, decoder‑only LLMs (3B, 8B, and 30B) trained on ~15T tokens using a multi‑stage pre‑training pipeline, including long‑context extension of up to 512K tokens.
Key facts
- The team use On-policy GRPO (Group Relative Policy Optimization) ( Shao et al., 2024 ) with DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) loss ( Yu et al
- Figure 10: The Granite 4.1 reinforcement learning pipeline consisting of four sequential stages: Multi-domain RL, RLHF, Identity and Knowledge-calibration RL, and Math RL
- Similar to the RLHF stage, they used a learning rate of 3e-7 and KL-loss coeff $\beta$ of 0.09, and they use 1728 unique prompts in this stage
- The team use an average of 13,504 unique prompts in this stage and similar to the multi-domain RL stage, they used a learning rate of 5e-7 and KL-loss coeff $\beta$ of 0.05
Summary
Building high‑quality small language models goes beyond simply scaling compute—it requires rigorous data curation throughout training. Granite 4.1 models use a decoder-only dense transformer architecture. All three model sizes share the same training pipeline and data strategy, differing only in architecture dimensions. Granite 4.1 is trained from scratch on approximately 15 trillion tokens using a five‑phase training strategy.