Fast, open-source language identification built for African languages.
If you’ve ever built a chatbot, moderation pipeline, or translation feature for Nigerian users, you’ve hit the same wall we did: before you can process text, you need to know what language it’s in — and most off-the-shelf language detectors fall apart the moment they meet Yoruba tone marks, Igbo diacritics, or Nigerian Pidgin’s blend with English.
Today we’re releasing the LID Collection: a family of open-source language identification models built specifically for this problem, available now on Hugging Face under the Apache 2.0 license.
Why language identification matters
Language ID (LID) is the invisible first step in almost every multilingual NLP pipeline. Route a Hausa message to an English-only model and everything downstream breaks — translation, sentiment, moderation, search. For African languages, this step has been chronically underserved: generic detectors either don’t support the languages at all, or confuse closely related ones (Pidgin vs. English is a notorious failure case).
The models
The collection covers a range of accuracy/size trade-offs so you can pick what fits your deployment:
LID-Neural-5 — our flagship. A transformer sequence classifier fine-tuned on castorini/afriberta_large (XLM-RoBERTa, 125M parameters), covering the five major languages of Nigeria: Yoruba, Igbo, Hausa, Nigerian Pidgin, and English. It achieves 98.96% macro accuracy with an average latency of just 13.3 ms per sentence. Per-language F1 scores range from 98.31% (English) to 99.60% (Yoruba) — and crucially, it handles the Pidgin/English boundary well.
LID-Neural-5.1 — a compact 30.9M-parameter variant of the 5-language model, for when you need the same coverage in a much smaller footprint.
LID-Neural-25.1 and 25.2 — expanded coverage across 25 languages, for multilingual applications that reach beyond Nigeria.
LID-Lite-5 and LID-Lite-25 — ultra-lightweight versions designed for edge devices, serverless functions, and anywhere GPU inference isn’t an option.
Getting started in one minute
The models are integrated directly into the olaverse Python library:
pip install olaverse[deeplearning]
from olaverse import LIDNeural5
detector = LIDNeural5()
detector.load()
detector.predict("Kedu ka ị mere today?") # → 'ibo'
detector.predict_proba("How far, wetin dey happen?")
# → {'eng': 0.002, 'hau': 0.001, 'ibo': 0.003, 'pcm': 0.991, 'yor': 0.003}
Prefer plain Hugging Face? Every model also works with a standard transformers pipeline:
from transformers import pipeline
lid = pipeline("text-classification", model="olaverse/lid-neural-5")
lid("Bawo ni, se daadaa ni?") # → [{'label': 'yor', 'score': 0.9987}]
You can also try the models with no setup at all in our live demo Space.
What’s next
The LID Collection sits alongside our other open releases — the MIST language models and DiacNet diacritic restoration models — as part of a broader goal: making African-language NLP a first-class citizen, not an afterthought. We’d love to hear where you deploy these models, what languages you’d like covered next, and where they fail. Open an issue on GitHub or start a discussion on any of the model pages.
Explore the collection: https://huggingface.co/collections/olaverse/lid
Olaverse Lab builds open-source AI for African languages. Follow us on Hugging Face, Twitter/X, and LinkedIn.