Models · Updated 16 Oct, 07:14 am IST
New 2.8T‑token Indic pretraining set and open GPU pipeline announced
Why it matters for readers: It gives anyone interested in Indian languages a documented way to create and inspect the data behind large language models for those languages.
- The release is Nemotron-pretraining-Indic-v1, approximately 2.8 trillion tokens covering 22 Indian languages, as described in the announcement.1
- The open GPU‑accelerated pipeline used to generate the data will be released and includes prompts, LLM workflows and quality assessment/repair steps.1
- The project is a collaboration between NVIDIA, IIT Madras and AI4Bharat and builds on prior open resources for Indic language research.1
- The 22 languages correspond to those listed in India’s Constitution and are described as first languages for about 97% of Indians per the 2011 Census.1
Get a brief like this every morning
Uzha reads hundreds of sources and gives you the stories that matter for your work, with every source linked. Free.
Get started