Open-Source

When developing open-source models and datasets for low-resource languages like Kinyarwanda, I consistently ran into the same architectural bottleneck: linguistic metadata is scattered everywhere. Whether you need ISO 639-3 codes, Glottolog family trees, or regional speaker counts, the data is typically siloed across mismatched CSVs, XML files, and SPARQL endpoints. Working as computational linguist with many languages, I needed a way to instantly access and cross-reference this global information without building custom data wrangling pipelines for every new NLP project.

In my daily work, I often need to process whole datasets with LLMs. This required my to always write the same code again. Many LLM providers like Google or OpenAI have special batch APIs, that reduce the token cost by 50%, but these was always too complicated to use. Therefore, I created Loom. Loom is a Python command line tool that makes batch processing easy.

The Large Language Models that we build are only as capable as the data they consume. For high-resource languages, finding massive, clean datasets is largely a solved problem. But when building models for low-resource languages, we kept hitting the same bottleneck: we needed a more agile way to crawl, extract, and structure web data to build our own training corpora. To solve this, I’m excited to share my latest open-source project: Crawlzilla.