Low-Resource Languages in AI: How to Collect Data From Scratch

There are more than 7,000 languages in the world, yet nearly half of the internet's biggest websites are written in just one — English. According to W3Techs, as of late 2025, English is used on 49.3% of the top 10 million websites. Language models learn from what's online, so they work great in English and noticeably worse in Swahili, Yoruba or Assamese, even though tens of millions of people speak them.

For teams bringing AI products to markets in Africa, South Asia and Southeast Asia, this is both a problem and an opportunity. The problem: there's no ready data. The opportunity: whoever collects it first gets a product that works better than the competition's. Below, we cover what data you need, the difficulties you'll run into, how to find native speakers and how to check quality even if you don't speak the language yourself.

What a Low-Resource Language Is

A low-resource language isn't necessarily a rare one. It's a language with little digital data: text, speech recordings, annotated datasets, dictionaries. Hundreds of millions of people speak Bengali, but there's many times less AI data per speaker than for German.

A 2020 Microsoft Research study shows the scale of the problem well. The authors split languages into six classes based on how much data is available:

By the authors' estimate, about a billion people speak class 0 languages. And there's another point: even languages with an impressive amount of text often lack exactly what a product needs — conversational speech, customer support dialogues, text on the right topics. For a business, any language your model handles noticeably worse than English is effectively "low-resource."

What Data You Need and Why

The type of data depends on what your model needs to do in the new language.

For most products, it makes sense to start with two things: a set of realistic prompts from native speakers, and ratings of your current model's responses to those prompts. That quickly shows you how bad things are and what to collect next. For voice products, see our guide to building a multilingual speech dataset; for text, see our article on text data collection for LLM fine-tuning.

Note: don't rely on machine-translating English data as your only source. For low-resource languages, machine translation is often inaccurate, and even an accurate translation sounds artificial and doesn't reflect how people actually write and speak. A model trained on such data will respond "like a translator," not like a native speaker.

The Difficulties You'll Run Into

Collecting data in a low-resource language only looks like collecting it in English from the outside. Here's what usually doesn't go to plan.

No standard spelling. In many languages, people write the same word in different ways: the rules haven't settled, and school was taught in another language. Decide up front what you need: the standard spelling or the way people actually write. For a chatbot, the latter often matters more.

Two scripts and transliteration. In Hindi, Arabic or Serbian, people often type in Latin script when messaging. If your product is a messenger or a support chat, collect both variants and state explicitly in the task which script to use.

Mixing languages. In India, the Philippines or Kenya, people constantly mix the local language with English in a single sentence. This is called code-switching. If you ban it in the task, you'll get textbook language that real users don't speak.

Dialects. One "language" can sound completely different in neighboring regions and countries. Arabic in Egypt and Morocco, Swahili in Kenya and Tanzania — these are different datasets. Record the dialect and region in the metadata of every entry.

Few speakers online. For some languages, there aren't many active internet users among native speakers, so collection is slower. Plan for more time and a higher reward than for popular languages.

No tools. Rare languages often lack reliable automated spell checking, language identification or even a keyboard with the right characters. So quality control has to be built on people — more on that below.

How to Find Native Speakers and Verify They're the Real Thing

The main risk when collecting data in a rare language is ending up with text run through a machine translator, or recordings from people who barely know the language. On the surface, this data looks fine, but it teaches the model the wrong language.

Where to look for native speakers

1. Microtask platforms with country targeting. Restrict the campaign to countries where most of the population speaks the language: Kenya and Tanzania for Swahili, the Philippines for Tagalog. It's the fastest way to recruit hundreds of workers.
2. Language communities and universities. Research communities like Masakhane for African languages are a good source of experts for complex tasks and review.
3. Professional translators. Expensive, but worth it for a small reference set you'll use to check everything else.

How to verify a native speaker

1. An entry task in the language. A short question to answer in their own words. Give the instructions for it in the language itself: someone who doesn't know it won't understand the task.
2. A local-knowledge question. For example, what a specific dish or object is called in their region. A machine translator won't help here.
3. A short audio recording. Ten seconds of free speech in the language immediately shows their level. Another native speaker reviews the recording.
4. A machine translation check. Run the task's source text through popular translators yourself. If a worker's answer nearly matches the machine output, that's a reason to reject the work.

Workers who pass verification and do well on their first tasks are worth bringing in to review others' work. This is how you build a small team of reliable reviewers for each language.

How to Control Quality When You Don't Speak the Language

This is a common situation: a team collects data in a language nobody on the team speaks. The solution is to build a process where native speakers check each other while you watch the numbers.

A two-stage "author and reviewer" setup. The first campaign collects the data; the second, with different workers, rates every entry: does it sound natural, are there errors, does it match the task. Even if you don't understand a word, you can see what the reviewers rejected and why.

Back-translation. For parallel data, ask a different native speaker to translate some sentences back into English. If the meaning drifts a lot, the original translation should be redone.

Control examples with a known answer. Order a small reference set from a professional translator and mix it into reviewers' tasks. This way, you're checking the checkers themselves.

Agreement between reviewers. Have two or three people rate some of the entries. If they often disagree, they may speak different dialects — in that case, split collection by region.

Simple automated checks. Even without knowing the language, you can catch duplicates, overly short answers, text in the wrong script and suspiciously close matches with machine translation.

Pro Tip: write the instructions in English using short, simple sentences, and have one of your verified native speakers translate them into the target language. Instructions in the worker's native language reduce errors and also filter out people who don't know the language.

What Data Collection Costs and How to Run It on RapidWorkers

Let's look at a typical first project: a team wants to launch a Swahili chat assistant for customers in Kenya and Tanzania. They need realistic prompts, translations of key phrases, review by native speakers and a small set of voice recordings (the rewards are for illustration — you set them yourself):

For languages with few speakers online, plan for higher rewards and more time to recruit: fair pay attracts more experienced people, while cheap tasks in a rare language can sit unfinished for a long time.

Here's how to launch:

1. Fund your balance by card or crypto, with a $20 minimum.
2. Run a screening campaign with an entry task in the language, restricted to countries where the language is widely spoken. RapidWorkers workers live in 190+ countries, including across Africa and South Asia.
3. Build a reviewer team from those who did well in screening, and ask one of them to translate your instructions.
4. Launch collection — a separate campaign for each data type and, if needed, for each dialect or country.
5. Run review of the collected data by your reviewers and accept work based on their ratings. Rejected work isn't paid.

We cover how to split collection across countries, and why a diverse pool of workers reduces model bias, in our guide to AI training data collection.

FAQ: Collecting Data for Low-Resource Languages

What is a low-resource language?

It's a language with little digital data: text, speech recordings, annotated datasets. Tens of millions of people may speak it, but because of the data gap, language models perform noticeably worse in it than in English.

Can I just machine-translate English data?

As a supplement, yes; as your only source, no. For low-resource languages, machine translation is often inaccurate and sounds artificial, and a model trained on it won't learn how people actually write and speak. At the very least, have native speakers review and correct such data.

Where can I find native speakers of a rare language?

The fastest route is a microtask platform with country targeting: restrict the campaign to countries where most people speak the language. Universities and language communities are useful for expert review, and professional translators for a reference set.

How do I check quality if I don't speak the language?

Build a process where native speakers check each other: some collect data, others rate it. Add back-translation for some entries, control examples from a professional translator and simple automated checks: duplicates, script, matches with machine translation.

How much data does a model need to work in a new language?

It depends on how well the base model already knows the language. Start with an evaluation: collect a few hundred realistic prompts and have native speakers rate the model's responses. The result will show whether you need thousands of examples for fine-tuning or whether adjusting prompts and adding a small set of examples is enough.

How much does data collection in a rare language cost?

In the example in this article, a starter dataset for a Swahili assistant — 2,000 prompts, 1,000 translated phrases, 3,000 reviews and 500 voice recordings — costs about $535 including the RapidWorkers fee. For languages with few speakers online, rewards and timelines are usually higher.

The Bottom Line: Data in Rare Languages Comes from Native Speakers, Not Translators

An AI product in a low-resource language runs into a data problem, not an algorithm problem, and that data can't simply be downloaded or translated from English. A simple approach works: clear rules for spelling, script and dialects, screening for genuine native speakers, and review in which speakers rate each other's work. Whoever collects this data first gains an edge in markets where competitors still answer customers with machine translation.

Start by screening native speakers and collecting a small set of prompts. Create a campaign on RapidWorkers, select the countries where the language is spoken, and the first responses will show you how far your model is from the way real people talk.