I Got Tired of Joining ISO 639-3 to Glottolog by Hand, So I Built low

If you have ever built a multilingual dataset or benchmark, you know the moment. You have a column of language codes - some ISO 639-1, some ISO 639-3, a few BCP-47 tags someone pasted in - and now you need to know a few simple things about each of them. Which family does it belong to? What script is it written in? Where is it spoken, and by how many people? Is it endangered?

None of these questions is hard. The answers all exist, in excellent open datasets. The problem is that they live in different datasets:

• SIL ISO 639-3 has the codes and reference names.
• Glottolog has the family tree and endangerment status - keyed by Glottocodes, not ISO codes.
• Unicode CLDR has official language status and per-country population shares - keyed by BCP-47 tags.
• Google’s LinguaMeta has speaker counts, scripts and names of languages in other languages.
• Wikidata, the CIA World Factbook and the UN M49 country list each add another piece.

So you download five files in four formats, write a join script, discover that half the codes don’t match, write a normalisation step, and an afternoon is gone. Then, three months later, on a different project, you do it again.

I have done this more times than I want to admit - most recently while working on an open LLM for African languages and on evaluation across hundreds of languages. So I finally did it once, properly, and packaged it.

Meet low

low (Languages of the World) is a small, read-only Python library that merges seven open datasets into one connected, in-memory object graph:

• 7,929 languages with ISO 639-3/639-1 codes
• 247 countries, grouped into UN regions and continents
• 4,853 Glottolog family nodes under 246 root families
• 106 writing systems (ISO 15924)
• ~9,000 per-country speaker counts, each tagged with its source
• ~168,000 canonical language names - endonyms and exonyms
• official, regional and de-facto official language status per country

The whole graph ships with the package, loads in under a second, and has no runtime dependencies.

pip install languages-of-the-world

A Quick Tour, Using Kinyarwanda

I have spent the last two years in Kigali, so let’s start with Kinyarwanda. One get() accepts whatever code or name you happen to have:

import low

db = low.LanguagesOfTheWorld()

kin = db.languages.get("rw")            # ISO 639-1
kin = db.languages.get("kin")           # ISO 639-3
kin = db.languages.get("Kinyarwanda")   # name, case-insensitive

print(kin.glottocode, kin.endangerment, kin.endonym.name)
# kiny1244 not_endangered Ikinyarwanda

Everything is connected, so you navigate with plain dot notation instead of writing joins. Here is Kinyarwanda’s full Glottolog lineage:

node = kin.family
while node:
    print("  " * node.depth + node.label)
    node = node.parent

                      West Highlands Kivu
                    Kivu
                  Western Lakes Bantu
                Great Lakes Bantu
              Northeast Savanna Bantu
            East Bantu
          Narrow Bantu
        Southern Bantoid
      Bantoid
    Benue-Congo
  Volta-Congo
Atlantic-Congo

And here is where it is spoken, according to each source:

for sc in kin.speaker_counts:
    print(f"{sc.country.label:40} {sc.speaker_count:>12,}  [{sc.source}]")

Congo, Democratic Republic of the             438,531  [cldr]
Rwanda                                     10,489,941  [cldr]
Rwanda                                      9,800,000  [linguameta]
Uganda                                      1,034,943  [cldr]

Notice that low does not pick a winner. Rwanda has two numbers because two sources gave two numbers. That is a deliberate design decision, and it leads to the most interesting part of this project.

What You See When You Put the Datasets Side by Side

Once everything lives in one graph, questions that used to take an afternoon become a few lines of code. The repository has ten example notebooks; here are three things that surprised me.

1. The sources mostly agree - until they really don’t

low has 985 country–language pairs where at least two of CLDR, LinguaMeta and the CIA Factbook report a speaker count. The good news: the median ratio between the highest and lowest estimate is 1.07 - usually they agree within a few percent.

The bad news is in the tail. 77 pairs differ by a factor of two or more, and 16 by a factor of ten or more. And the biggest disagreement in the entire dataset happens to be right here, in Rwanda:

for sc in db.speaker_counts.for_country("RW"):
    if sc.language.part3 == "fra":
        print(sc.source, f"{sc.speaker_count:,}")
# cldr        790,151
# linguameta    2,300

French in Rwanda: roughly 790,000 speakers according to CLDR, 2,300 according to LinguaMeta. A factor of 344.

Neither number is a typo. They are answering different questions. CLDR’s figure reflects people who can use French - a legacy of the decades when French was a main language of education and administration. LinguaMeta’s figure is closer to people who speak it as a first language. Both are defensible. But if your pipeline silently takes one of them, you are making a methodological decision without knowing it. Other examples in the top of the list follow the same pattern: French in Djibouti and the DR Congo, Spanish in Paraguay, Piemontese in Italy.

This is why low keeps every source’s estimate next to each other, labelled. If you build sampling weights, “low-resource” thresholds or population-weighted metrics on top of speaker counts, look at the disagreement before you trust any single number.

2. The official language is often not the one people speak

Using CLDR’s population shares and official-status flags, you can ask: in which countries is the most widely spoken language not an official language?

for country in db.countries:
    official = {l.part3 for l in country.official_languages}
    cldr = [sc for sc in country.speaker_counts if sc.source == "cldr"]
    if not official or not cldr:
        continue
    top = max(cldr, key=lambda sc: sc.speaker_count)
    if top.language.part3 not in official:
        print(f"{country.label}: {top.language.label} ({top.speaker_fraction:.0%})")

That gives 27 countries. Some highlights:

The list is dominated by West Africa and by creoles - languages spoken by almost everyone in a country, yet absent from its official status and, not coincidentally, underrepresented in NLP training data. If you are deciding which languages to prioritise for a model or a benchmark, “official languages of country X” is a poor proxy for “languages people in country X actually use”.

The list also contains a group of Arabic-speaking countries - Morocco, Tunisia, Algeria, Lebanon - where the top language is a spoken variety (Moroccan Arabic, Tunisian Arabic, …) while Standard Arabic is official. That is real diglossia, but it is also partly an artefact of how ISO 639-3 splits the Arabic macrolanguage. Data like this always needs a human reading it.

3. Half of the world’s assessed languages are endangered

Glottolog assigns an endangerment status to 7,524 of the languages in low. Of those, 3,923 living languages are threatened, shifting, moribund or nearly extinct - more than half - and another 835 are already extinct.

Because every language knows its countries and every country knows its continent, the regional breakdown is one loop away. Share of living, assessed languages spoken on each continent that are endangered:

Africa has the lowest share - not because its languages are safe, but because many African languages still have large, growing speaker communities. That is exactly why the gap between how many people speak these languages and how little digital data exists for them matters so much.

Caveats, Honestly

low is a convenience layer, not a new source of truth. It inherits every bias and gap of its upstream data, and it makes a few choices of its own:

• Speaker counts are estimates. They are often rounded, sometimes outdated, and - as shown above - they don’t all measure the same thing. Language.speaker_count takes the maximum of LinguaMeta and Wikidata; if you need something more careful, use the per-source speaker_counts.
• CLDR counts second-language users. That is why, for example, English shows up as the most widely spoken language in Belgium. Know which source you are reading.
• Codes are hard. Matching BCP-47 tags, ISO 639-1/-3 codes and country names across sources works for the vast majority of entries, but macrolanguages and renamed countries will occasionally surprise you.
• There is an optional scraper. For country–language pairs where no source has a number, low can fill gaps with counts extracted from web pages by an LLM. These are tagged source="scraped", never overwrite the curated sources, and should be treated as lower-confidence.

The code is MIT-licensed; the bundled data keeps the licences of its upstream sources (mostly CC BY 4.0 and CC0), so please credit them if you redistribute it.

Who It’s For

• NLP researchers building multilingual datasets, benchmarks or language-coverage reports - the people I built it for.
• Linguists and educators who want to explore families, scripts and endangerment without a database setup.
• Data journalists who need defensible numbers about who speaks what, and where.
• Developers who just need a correct, complete list of languages with names and codes.

Try It, Break It, Improve It

pip install languages-of-the-world

• Code, documentation and the ten example notebooks: github.com/jnehring/languages-of-the-world
• Package: pypi.org/project/languages-of-the-world

If it saves you an afternoon, a star on GitHub helps others find it. If you find a wrong number, a missing language or a mismatched code, please open an issue - and if you know of another open dataset that belongs in the graph, I would love a pull request.