Manual Data Verification: How to Check and Clean Data with Human Reviewers

In 2021, MIT researchers audited the test sets of ten of the most widely used machine learning datasets, ImageNet among them, and found an average label error rate of at least 3.3%. In the ImageNet validation set, about 6% of labels were wrong: 2,916 images. How they found those errors is the more interesting part. An algorithm flagged suspicious examples, and people made the final call. Each questionable label was shown to five Mechanical Turk workers, and an error counted as confirmed only on their votes. Of everything the algorithm flagged as mislabeled, humans confirmed roughly half.

That's manual data verification in practice: automation narrows the field, and a person makes the decision. Below, we cover which data is worth sending to human reviewers, how to write verification rules, how many reviewers each record needs, how to tell whether the reviewers themselves are getting it right, and what it all costs.

How Manual Verification Differs from Script-Based Validation

Automated validation checks the shape of the data: is there a value in the field, does the email look like an email, is the price within the allowed range. Manual verification answers a different question: does the record match reality? An address can be perfectly formatted while the café at that address closed a year ago. The label "cat" can sit in the right column while the photo shows a dog.

So the question isn't which one to choose, but in what order. First, a script removes everything a rule can catch, so people don't waste time on empty rows and broken formats. People get what has to be compared against the real world.

What Data Is Worth Verifying Manually

Bad data is expensive: Gartner estimates it costs organizations an average of $12.9 million a year. But there's no point in checking everything by hand. People are needed where a record's correctness can't be inferred from the record itself. Most often, that means five types of data:

1. AI training labels. Does the label match what's in the image, text or recording? This is the most common use case: some share of labels from any source is worth re-checking by other people.
2. Scraped product catalogs. Does the title match the photo, the price match the price on the site, the category match the product? Scrapers often grab a neighboring block or an old version of the page.
3. Business listings. Does the business exist, is it open right now, do the name, address and opening hours match? For directories and delivery services, this is the main source of complaints.
4. Web data. Does the link work, and does the page actually contain the information stored in your database? A script sees a 200 status code but not that the domain was sold long ago and now shows a parked page.
5. On-site checks. A person in the right city walks to a public place and sends a photo: is the store there, is the product on the shelf, is the sign up? This requires geo-targeting, or the task will be picked up by people in other countries.

Note: manual verification is for public information about businesses, products and places, and for your own datasets without personal data. Checking up on private individuals (looking them up, calling them or visiting them) is not allowed through microtasks.

How to Write a Data Verification Task

Many reviewer errors come not from carelessness but from different ideas of what "correct" means. If the price on the site is $19.99 and the database says $20, one person will mark it "correct" and another "wrong." Both have a point. That's why a good task is built from six parts:

1. A small, numbered batch. 5–15 records per task. Fewer isn't worth it for you or the worker; more, and attention drops toward the end.
2. A source to compare against. What exactly to check the record against: the product page at the link, the image itself, a map, the company's official website. "Search online" produces ten different sources and ten different answers.
3. Three statuses with definitions. "Verified," "Wrong" and "Unclear," each with a written rule for when to use it. Without a third status, people guess, and those guesses are impossible to tell apart from confident answers.
4. Edge cases. Is a price difference under 1% still "verified"? A name in Latin script instead of Cyrillic? A store closed for renovation? Decide in advance and write it down.
5. A note or proof for every error. Briefly: what didn't match and where it's visible (a link, a screenshot, a photo). Notes make your review much faster.
6. An answer format. One line per record: number, status, note. That makes it easy to merge answers from different people into one table.

Example task: "Below are 10 products with links. Open each link and compare the title, price and photo with the data in the table. VERIFIED: everything matches and the price differs by less than 1%. WRONG: at least one field doesn't match; write which one. UNCLEAR: the page doesn't load or the product is out of stock. Answer with one line per product in the format 'number — status — note'."

Pro Tip: before launch, check one batch yourself and time it. You'll immediately spot edge cases the rules don't cover, and you'll know what a fair rate per task is.

How Many Reviewers Each Record Needs

One person is cheaper, but their mistakes go unnoticed. With two or more people checking the same record independently, disagreements show you where the problems are. That's exactly how the MIT researchers worked: five votes on each questionable label, with errors confirmed by majority. How many people you need depends on what an error costs you (typical practice, not hard rules):

Treat the test set with particular care. It's what you judge your model's quality by, and the same researchers showed that after fixing test set errors, model rankings can change. Three reviewers per record almost always pay off there. We wrote more about building these checks into a workflow in our article on human-in-the-loop AI.

How to Tell Whether Reviewers Are Getting It Right

Manual verification produces errors too, and you need to measure them rather than hope for the best. There are three tools for this, and they work best together.

Control records. Add 1–2 records with a known answer to each batch, including deliberately wrong ones. If someone marks a planted error as "verified," they most likely set statuses without looking. Reject those batches in full.

Agreement between reviewers. When two people check a record, track how often they agree. Low agreement across the whole set usually means the rules are vague, not that the people are bad. Low agreement between one particular person and everyone else is a reason to re-check their work.

Spot-check review. Check some batches yourself. That gives you the real accuracy of the verification, not an impression. For more quality control practices, see our article on managing microworker output.

Warning: don't reject work just because someone disagreed with the majority. Sometimes they're the one who's right, especially if they attached a screenshot. Reject for failed control records and broken format, and always give a specific reason.

Algorithm First, Human Second: How Not to Over-Verify

With hundreds of thousands of records, even cheap verification of every single one adds up. A workflow that works looks like this:

1. A script removes duplicates, empty fields and broken formats.
2. An algorithm flags suspicious records. For labels, that can be the open-source library cleanlab, built on the method from the same MIT study. For a catalog, it's a price that differs sharply from similar products.
3. People check the flagged records plus a random sample of the rest.
4. The random sample tells you how many errors the algorithm missed.

Step 3 matters more than it seems. The algorithm is wrong in both directions: in the MIT study, humans confirmed only about half of its suspicions. If you automatically delete everything it flags, you'll lose hard but correct examples along with the errors, and those are exactly the ones your model needs.

What Not to Send to Microtasks

Crowdsourced manual verification has clear limits:

1. Personal and confidential data. Customer databases, contacts of private individuals, internal documents. Strip everything personal before uploading, or don't upload that data at all.
2. Decisions that require a profession. Whether a medical diagnosis, a legal classification or a technical specification is correct should be judged by a specialist.
3. Split-second decisions. Real-time moderation or payment checks at checkout. Microtasks take hours, not milliseconds.
4. Checking up on people. Finding, calling or photographing a private individual is not allowed under any pretext. On-site checks happen in public places only.

On-Site Checks: When You Need a Person in a Specific City

Sometimes there's no reliable source online. The map listing says "open," the company website hasn't been updated since last year, and the reviews contradict each other. In those cases, the only reliable check is a person who walked to the spot and photographed it.

Typical tasks: is a retail point from a distributor's list still operating, is your product on the shelf, does the sign match the name in the catalog, is there an entrance at the listed address. These tasks have their own rules:

1. One location, one task. Getting there takes longer than the check itself, so the rate here is several times higher than for an online check.
2. A photo as proof. The frame should include the sign or another recognizable feature of the place, not just a wall or a shelf.
3. No close-ups of people. The photo is of the place, not of staff or shoppers.
4. Public places only. Stores, streets, building entrances. Entering restricted areas or taking photos where it's prohibited is not allowed.

Manual Data Verification on RapidWorkers: Process and Cost

On RapidWorkers, data verification runs as a regular microtask campaign. You upload a batch of 5–15 rows, describe the rules and statuses, and choose the countries and the rate. More than 195,000 workers from 200+ countries are registered on the platform, so even for an on-site check in a specific city, there are usually people nearby. A worker submits one answer per batch, and you approve or reject it. Formats and the self-serve and fully managed options are described on the manual data verification page.

The employer sets the rate per task, and a 15% platform fee is added on top. The minimum deposit is $20 by card or crypto, and the minimum campaign is $0.30. Rejected work isn't paid, and if you end a campaign early, the unused budget goes back to your balance (a $0.20 cancellation fee may apply). Here's an example calculation with illustrative rates for verifying 1,000 records in batches of 10:

A double check costs twice as much, but it often pays for itself: you stop reviewing every record yourself and only resolve disagreements. And if the rate is too low for the amount of work, people start setting statuses without looking, and the savings turn into rework.

Manual Data Verification FAQ

What is manual data verification?

It's checking records against a source or the real world, done by a person: does the label match the image, the price match the website, the address match an operating business. It complements automated validation, which only checks format and completeness.

How do you verify data manually without spending weeks on it?

First, use a script to remove formal errors. Then split what's left into batches of 5–15 records and hand them out to different people with clear rules. A hundred people working in parallel will check in a few hours what would take one employee weeks.

How many people should verify each record?

For most tasks, one reviewer plus a spot-check is enough. For data your customers will see or a model will be trained on, use two. For a test set, use three or more and decide by majority.

Can you trust data validation by humans you don't know?

Yes, as long as the trust is measured. Control records with known answers, comparing answers across reviewers and spot-check reviews show you the real accuracy. Work only goes into the clean file after you approve it.

How much does manual data verification cost?

On RapidWorkers, you set the rate per task, plus a 15% fee. In the example above, verifying 1,000 records in batches of 10 at $0.30 costs $34.50, and a double check of the same volume costs $69. On-site checks cost more because the person has to travel to the location.

What Makes Manual Data Verification Useful Rather Than Expensive

Manual verification pays off where a script is powerless: labels, catalogs, business listings, links and real places. Results depend on four things: automation filters out formal errors first, every status has a definition, the number of reviewers matches the cost of an error, and the reviewers' own accuracy is measured with control records. Start with a 100-record pilot: it will expose the weak spots in your rules before you pay for thousands.

If you need to verify a dataset, a catalog or a list of locations in specific countries, launch a pilot campaign on RapidWorkers. We covered the overall process of AI data collection in our guide to AI training data collection.