Open Source · India · Hugging Face
Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
These are the 22 languages listed in India's Constitution: the first languages for 97% of Indians, or nearly 1.2 billion citizens at the 2011 Census.
Key facts
- Figure 4 compares Nemotron 3.5 Super VL with Nemotron 3 Super across Indic task performance and language quality — These are the 22 languages listed in India's Constitution: the first languages for 97% of Indians, or nearly 1.2 billion citizens at the 2011 Census
- The configured budget was 100 billion training tokens, with 10% (10 billion tokens) allocated to a mixture of synthetic and natural Indic text
- Continued pretraining for Nemotron 3.5 Super VL added 10 billion Indic tokens, drawn from a twelve-language subset of this corpus and natural Indic text
Summary
They're excited to introduce Nemotron-pretraining-Indic-v1 approximately 2.8 trillion tokens across 22 Indian languages, releasing soon alongside the open, GPU-accelerated data generation pipeline used to create it. AI4Bharat, IIT Madras' Indian language research lab, builds open-source datasets and models for Indian languages. Its resources, such as Sangraha, an open pretraining corpus for 22 Indian languages, have helped lay the foundation for open Indic language-model research. Mitesh Khapra, who heads AI4Bharat and made the 2025 TIME100 AI list, has named the core problem: Indian language technology trails English because "They do not have enough data for Indian languages. The corpus also accounts for how people write. They're sharing the open generation recipe so the community can not reproduce the data but take this work further.