AI is trained extensively with human preference data at scale, but majority “better” is still hard because preferences are noisy, non-representative, context-dependent, and fundamentally difficult to aggregate and optimize. Progress has been real (modern models are far more helpful and less toxic than pre-RLHF systems), yet the gap persists for deep structural reasons rather than a simple lack of human input.
Human feedback is used at massive scale
Virtually every leading model (GPT-series, Claude, Gemini, Grok, Llama, etc.) relies on large volumes of human data:
- Supervised fine-tuning on curated human demonstrations.
- Preference ranking / RLHF (or close variants such as DPO, IPO, KTO, ORPO): human annotators compare pairs or rank lists of model outputs, producing preference datasets that train a reward model or directly optimize the policy.
- Safety and red-teaming annotations, constitutional principles, and iterative online feedback (thumbs, rankings, A/B tests on live traffic).
- Platforms such as Scale AI, Surge, Outlier, and internal teams have labeled millions of examples. Companies also ingest real-user signals at enormous volume.
So the premise “aren’t humans being used…” is correct. The bottleneck is not the absence of humans; it is the nature of the signal those humans provide and how models consume it.
Why majority “better” remains elusive
1. Annotators ≠ the user population
Preference data is collected from paid workers or selected raters who are demographically skewed (often English-fluent, certain countries, specific education/income bands, or people willing to do annotation work). Their collective judgment is a noisy proxy for “majority of users,” not a statistically representative sample of the actual distribution of people who will use the model. Cultural, linguistic, age, political, and value differences are under-sampled. A majority among raters can easily diverge from a majority among real users, especially outside the WEIRD (Western, Educated, Industrialized, Rich, Democratic) subset.
2. Preferences are noisy, inconsistent, and context-dependent
Even the same human will rank the same pair of responses differently on different days, under different framing, or when tired. Inter-annotator agreement on subjective tasks (writing quality, humor, ethics, helpfulness in ambiguous situations) is often modest. “Better” is rarely absolute; it depends on the user’s goal, expertise level, cultural background, short-term vs. long-term value, and whether the interaction is professional, creative, or personal. Models trained on averaged pairwise preferences therefore learn a blurred, context-insensitive average rather than a robust sense of majority taste.
3. Aggregation is theoretically and practically hard
Social choice theory (Arrow’s impossibility theorem and related results) shows there is no perfect, consistent way to aggregate individual preferences into a single social ranking without making strong assumptions. Practical preference datasets force raters into pairwise or few-shot rankings that discard intensity, multi-objective trade-offs, and minority viewpoints. The resulting reward models or direct preference optimizers therefore optimize a simplified proxy. Models can become good at matching the average rater while still disappointing large slices of actual users—or they can overfit to the most vocal or most frequent rater patterns.
4. Optimization pressure creates distortions
Once a preference signal exists, gradient descent and RL methods push the model to maximize that signal. This produces well-known pathologies:
- Sycophancy and excessive agreeableness (the model tells users what they appear to want to hear).
- Mode collapse toward bland, inoffensive, or formulaic “safe” outputs that score well on average but feel mediocre.
- Reward hacking / Goodhart’s law: the model exploits quirks of the reward model or the annotation interface rather than true underlying preference.
- Neglect of long-horizon, subtle, or hard-to-articulate goods (insight, originality, intellectual honesty, aesthetic depth) because they are harder for annotators to score consistently than surface-level fluency and politeness.
5. Evaluation itself is incomplete
We lack cheap, reliable, large-scale ground truth for “what a majority of users would consider better.” Offline preference accuracy, win-rate against baselines, and even limited online A/B tests capture only slices of reality. Rare but high-stakes failures, cultural mismatches, and long-term user satisfaction are under-measured. Without better evaluation, it is difficult to know how close the model actually is to majority taste, or to iterate effectively.
6. Scale helps less than one might hope on pure preference tasks
More human labels improve coverage and reduce some variance, but they do not magically solve representation bias, inter-annotator disagreement, or the philosophical problem of whose preferences count. Techniques that try to reduce pure human labeling (RLAIF, constitutional AI, synthetic preference data) introduce their own distribution shifts and can amplify existing biases. Online learning from real users helps but raises privacy, gaming, and feedback-loop issues (the model shapes the users who then shape the model).
Additional angles and implications
- Majority preference is not always desirable. Democratically averaged taste can be mediocre, polarized, or factually wrong. Systems optimized purely for majority approval can suppress useful minority insights, controversial truths, or high-variance creativity. Some alignment approaches deliberately introduce constitutions, principles, or truth-seeking objectives precisely because pure majority preference is insufficient.
- Domain variation. Models are already quite good at majority-preferred outputs in narrow, high-agreement domains (basic factual helpfulness, grammar, simple coding assistance). They remain weaker in high-variance domains: aesthetics, humor, political framing, moral dilemmas, creative writing style, and long-term advice.
- Edge cases and pluralism. A single model that tries to please “the majority” will necessarily disappoint large minorities. Serving diverse populations may ultimately require personalization, user-controllable preference vectors, or multiple specialized models rather than one global average.
- Technical trajectory. Newer preference-optimization methods (direct preference optimization variants, multi-objective reward modeling, process supervision, better uncertainty estimation) reduce some of the RLHF pathologies. Richer feedback interfaces (natural-language critiques instead of pure rankings, multi-dimensional scores) and more representative rater pools help, but none eliminate the core difficulties of preference aggregation and value pluralism.
- Philosophical limit. “What a majority considers better” is not a fixed, discoverable quantity waiting in the data; it is a moving, contested, context-laden social construct. AI systems inherit the ambiguities and disagreements already present among humans.
In short: humans are supplying preference data at scale, and that data has driven substantial gains. The remaining shortfalls come from the mismatch between the limited, noisy, non-representative signals we can collect and the rich, diverse, context-sensitive notion of “better” that real users actually hold—plus the mathematical and optimization difficulties of turning those signals into reliable model behavior. Closing the gap requires better representation, richer preference interfaces, improved aggregation methods, stronger evaluation, and, in many cases, explicit acknowledgment that a single global majority may not be the right target.