Crawlzilla

The Large Language Models that we build are only as capable as the data they consume. For high-resource languages, finding massive, clean datasets is largely a solved problem. But when building models for low-resource languages, we kept hitting the same bottleneck: we needed a more agile way to crawl, extract, and structure web data to build our own training corpora. To solve this, I’m excited to share my latest open-source project: Crawlzilla.

Crawlzilla is a lightweight, adaptable web crawling and data aggregation tool. Whether you are scraping linguistic metadata, assembling parallel corpora for machine translation, or pulling together text from diverse regional sources for an LLM dataset, Crawlzilla handles the heavy lifting of data extraction.

What makes Crawlzilla special to me isn’t just the codebase—it’s how it was built. I developed Crawlzilla in close collaboration with an incredibly talented community of open-source developers here in Rwanda. Through my work mentoring local technology teams and collaborating within the Mbaza open-source AI community in Kigali, we’ve been tackling the unique infrastructure challenges of localized AI. Collaborating across borders and skill levels to build this tool has been a highlight of my time working in the Rwandan tech ecosystem.

If you are working on NLP research, building local language datasets, or just need a flexible crawler for your next project, check out the repository:

github.com/jnehring/crawlzilla

Let’s keep building open tools together.