RLHF Data Explained: How Human Feedback Is Collected for AI

In 2022, OpenAI published its InstructGPT paper with a result that surprised many: people preferred the outputs of a 1.3-billion-parameter model to those of the 175-billion-parameter GPT-3, even though the first was 100 times smaller. The difference didn't come from a new architecture or more compute. It came from data: responses written by people and thousands of human judgments of the form "this response is better than that one."

This approach is called RLHF — reinforcement learning from human feedback. Today, it and its simplified variants are used not only by large labs but also by teams fine-tuning open models for their own products: for a brand's tone, a specific domain or a particular language. The algorithms are available in open-source libraries. The bottleneck is almost always the data.

What RLHF Is, in Plain Terms

Classic RLHF has three stages, and two of them need people:

1. Supervised fine-tuning (SFT). People write model answers to prompts, and the model learns to imitate them.
2. Reward model. The model generates several responses to the same prompt, and people compare them and pick the better ones. These comparisons train a separate model that learns to predict which response a person will prefer.
3. Reinforcement learning. The main model generates responses, the reward model scores them, and the main model gradually shifts toward high-scoring responses. People are no longer involved at this stage.

In 2023, a method called DPO (Direct Preference Optimization) appeared. It skips the separate reward model and trains the model directly on "better vs. worse" pairs. It's simpler and cheaper to run, so it has become popular with teams fine-tuning open models. But for the purposes of this article, something else matters more: classic RLHF and DPO need the same raw material — human preference data.

Note: don't confuse RLHF with human evaluation of models. Evaluation answers the question "how good is the model right now," while RLHF data is used to change it. The tasks people do look similar, but the requirements for data volume and variety are different. We covered evaluation in detail in our article on human evaluation of LLM outputs.

Types of Human Feedback Data

"RLHF data" usually means preference pairs, but in practice teams collect several kinds of feedback. Each has its own purpose and cost.

For most teams, the main format is preference pairs. They're the easiest to collect, they work for both classic RLHF and DPO, and it's easier for a person to compare two texts than to give one text an exact score.

What a single record looks like

A preference pair is a prompt plus two responses: the chosen one and the rejected one. Public datasets like Anthropic HH-RLHF are stored in this format, and popular DPO libraries expect it too. A simplified record looks like this:

It's worth storing metadata alongside: who rated the pair, how confident they were ("much better" or "slightly better"), a short reason for the choice and the prompt category. This helps you filter out weak pairs and spot biases in the data later.

How Preference Data Is Collected: Step by Step

Step 1. Gather prompts

Prompts determine what the model will learn. If your set contains only coding questions, the model won't get any better at handling customer complaints. Good sources:

1. real user prompts from your logs (with personal data removed and with user consent);
2. prompts written on request by people from your target audience: "write a question you would ask a bank's assistant";
3. hard and borderline cases you add yourself: ambiguous questions, requests the model should politely decline, long multi-step instructions.

Having people write prompts is a separate and inexpensive task. It's how you get natural phrasing in different languages, with typos and colloquialisms, just like real users. For more on collecting text from people, see our guide to text data collection for LLM fine-tuning.

Step 2. Generate candidates

For each prompt, get two or more responses. These usually come from the current version of the model with different generation settings, or from several different models. The responses need to differ noticeably: if they're nearly identical, the person picks at random, and the pair teaches the model nothing.

Step 3. Write guidelines for labelers

The guidelines are the most important part of the project, because they define which values your model will ultimately absorb. They should include:

1. the selection criteria and their priority: for example, safety first, then accuracy, then helpfulness, then style;
2. what to do when criteria conflict: a short accurate answer versus a long, detailed one;
3. 5–10 worked examples with an explanation of the choice;
4. a rule for when both responses are bad.

Step 4. Pilot and calibrate

Before the main collection, give 50–100 pairs to several labelers and compare their choices with yours. Where they often disagree with you, the guidelines are unclear or incomplete. Fill the gaps and repeat the pilot.

Step 5. Main collection with quality control

Split the pairs into batches and add control pairs with an obvious answer to each one. Give some pairs to several labelers so you can measure agreement. Check time on task and read the explanations for each choice.

Step 6. Clean and assemble the dataset

Remove pairs from careless labelers, pairs marked as ties and pairs where labelers disagreed. Check the balance: the chosen response shouldn't almost always be the longer one, and prompt categories should be represented in the proportions you need.

How Much Preference Data You Need

There's no universal number, but there are clear reference points. Here's how much data well-known public projects used:

Note the rankings in InstructGPT. A single ranking of K responses yields K×(K−1)/2 pairs: 6 pairs for four responses, 36 for nine. So ranking lets you collect many comparisons per task, although the task itself is noticeably harder.

A team fine-tuning an existing open model for a narrow task usually doesn't need volumes like these. The base model already follows instructions, and preference data just nudges it in the right direction. Practical reference points (these are starting points, not strict rules):

An iterative approach works best: collect a first batch, fine-tune the model, evaluate the result and see where it still makes mistakes. Collect the second batch on the new version's responses, focusing on the weak spots. For example, Anthropic collected part of the HH-RLHF data exactly this way: on responses from already fine-tuned models, round after round.

Pro Tip: quality beats quantity. A thousand clean pairs with a clear difference between responses will give you more than ten thousand pairs where labelers chose at random.

What Ruins Preference Data and How to Avoid It

The model doesn't learn what you meant. It learns what labelers actually chose. Any systematic bias in their choices, the model will amplify. Here are the most common problems.

Preference for longer responses. A long response often looks more thorough, even if it's no more useful. A 2023 study showed that a noticeable share of the "improvements" after RLHF came simply from responses getting longer. State it plainly in your guidelines — "don't pick a response just because it's longer" — and after collection, check how often the longer response wins.

Sycophancy. People like being agreed with. In Anthropic's study on sycophancy, both humans and reward models sometimes preferred a convincingly written response that agreed with the user over a correct one. Include prompts with false premises in your set and give labelers a rule: politely correcting a mistake is better than agreeing with it.

Confident tone instead of accuracy. A labeler without subject knowledge can't tell a confident fabrication from a fact. For factual questions, include reference material in the task or send those pairs to people with the right background.

Responses that are too similar. If two responses differ by a couple of words, the choice is random, and random labels are noise for training. Add a confidence scale ("much better / slightly better / about the same") and leave pairs in the last category out of training.

A homogeneous group of labelers. Ideas of what makes a response polite, appropriate and helpful depend on culture. If all your labelers are from one country, the model will absorb that country's norms. For a product serving several markets, recruit labelers from each of them. We cover this in more detail in our article on reducing dataset bias with contributors from different countries.

Fatigue and careless work. Attention drops toward the end of a long session, and some workers choose at random from the start. The fix: short batches of 10–15 pairs, control pairs in every batch, and no payment for work that fails the controls.

What RLHF Data Collection Costs and How to Run It on RapidWorkers

Preference data collection breaks down neatly into microtasks: write a few prompts, compare a batch of response pairs, fix a weak response. On RapidWorkers, each of these stages is a separate campaign. The worker opens a link to your form or labeling tool (Google Forms, Label Studio or your own interface), completes the task and submits a completion code. You review the work and pay only for what you accept.

Here's an approximate budget for a first batch of DPO data (the rewards are for illustration — you set them yourself):

Generating the model's responses isn't included: that's your own compute or API calls. Demonstrations, where a person writes a model answer from scratch, cost noticeably more than comparisons: one good written answer takes about as long as a dozen comparisons.

Here's how to launch:

1. Fund your balance by card or crypto, with a $20 minimum.
2. Create a prompt-writing campaign if your own logs aren't enough. Select your users' countries so the phrasing sounds natural.
3. Generate response pairs and load them into your labeling tool in batches of 10, with control pairs in each batch.
4. Run a pilot of 10–20 tasks and compare the workers' choices with your own. Refine the guidelines where they disagree.
5. Scale up and accept work as it comes in. Rejected work isn't paid.

Note: don't show workers your customers' confidential data. If your prompts come from real conversations, remove names, phone numbers, addresses and other personal data before launching the campaign.

FAQ: RLHF Data

What is RLHF data?

It's a record of human judgments about a model's responses. Most often it takes the form of preference pairs: a prompt, the response the person judged better and the response they rejected. It also includes model answers written by people, rankings of several responses and scale ratings.

How is RLHF different from DPO?

Classic RLHF first trains a separate reward model and then fine-tunes the main model with reinforcement learning. DPO skips the reward model and trains the model directly on preference pairs, which makes it simpler and cheaper to run. Both methods use the same data: "better vs. worse" pairs.

How many preference pairs do I need for fine-tuning?

To adapt an existing open model to a narrow task, teams often start with 1,000–10,000 pairs, depending on how complex the domain is. For comparison, Anthropic's public HH-RLHF dataset contains about 169,000 pairs, but it was built for a general-purpose assistant. It's better to collect data in several rounds and evaluate the model after each one.

Can I use public datasets instead of my own?

For general skills, yes, and teams often use them as a foundation. But public datasets don't know your product, your brand's tone or the rules of your domain, and they're almost always in English. Check the dataset's license before using it, too.

Who should label preference data?

People who resemble your users, for judging helpfulness, clarity and tone. Native speakers, for multilingual models. Specialists only where the choice requires professional knowledge: medicine, law, complex code.

Can another model's ratings replace human labelers?

Partly. Using automated ratings is called RLAIF, and it's cheaper. But the rating model passes its own biases on to the model being trained, and without a human reference set you can't check whether it rates correctly. A mixed approach works: people label the hard cases and a control sample, and the model handles the simple ones.

The Bottom Line: Your Model Will Be as Good as Your Preference Data

RLHF and DPO algorithms are available in open-source libraries, and base models have become strong and accessible. What separates an average product from a good one today is the data: how closely the prompts resemble real ones, how clear the labeler guidelines are and how carefully random and biased labels have been filtered out.

A first batch of a few thousand pairs costs hundreds of dollars, not tens of thousands, and takes days, not months. Start with a pilot: write your guidelines, prepare 100 pairs and create a campaign on RapidWorkers with workers from your users' countries. The first results will show you where the guidelines work and where they need more work.