Can AI Synthetic Respondents Replace Market Research? Pew’s Digital Twins Missed by 12 Points

Asking an AI to "act as 30 women in their 30s and pick a landing page" takes ten minutes; a real survey takes weeks. That speed is why AI synthetic respondents are spreading fast in market research. A new experiment from Pew Research Center, published Sept. 30, shows why marketers should be careful with the numbers they produce.

Why researchers are turning to AI

• In a late-2024 Qualtrics survey of about 3,000 market research professionals across 14 countries, 71% agreed most market research would use synthetic responses within three years; 87% of those who had tried them were satisfied.
• Korean coverage reports consumer studies that took four to six weeks can be done in two to three days.
• Academic work raised expectations: Stanford's AI agents built from two-hour interviews with 1,052 Americans reached 82–86% accuracy relative to people's own consistency, and a 2026 Nature study predicted message effects across 70 experiments with a 0.85 correlation.

How Pew tested digital twins

Pew built an AI "digital twin" for each member of its American Trends Panel, then replicated three 2026 surveys of 8,512, 3,507 and 5,103 people — nearly 300 questions in total. Claude Opus 4.6 had the lowest error in a qualifying round and was used for the main test.

The result: a 12.4-point average error

Mean absolute error averaged 12.4 percentage points across the three surveys, and 28% of questions missed by more than 15 points. A typical national poll claims a margin of error around ±3 points.

Five ways synthetic respondents go wrong

1. They rarely say "not sure"

Humans picked "not sure" 16% of the time; the twins picked it 4%. On a civics question about the First Amendment, 52% of people answered correctly versus 98% of the AI.

2. They turn majorities into unanimity

Real Hispanic adults said 42% would likely watch the next World Cup; their twins said 97%. Errors were largest for smaller and more partisan groups. The model answers with a stereotype of the group, not the person.

3. They miss recent news

After the model's knowledge cutoff, errors jumped. 52% of people were very worried about electricity costs; the AI said 23% — the largest gap, at 29 points.

4. Results depend on the model

Given the same profiles, GPT-5.1 produced more extreme opinions while Opus produced more moderate ones.

5. Clean-looking data is dangerous

Bad synthetic data still produces tidy tables and charts. Pew concluded AI models are not an adequate substitute for traditional polling, though it endorsed AI for coding open-ended answers and analysis.

Vendors list the same limits

Korean firm Macromill Embrain found small human-AI gaps on concept and naming tests but significant gaps on brand awareness and real purchase behavior. Opensurvey warned about variance collapse, social desirability bias, Western and highly educated training data, and poor reproducibility. Qualtrics advises against using synthetic data for launch decisions, major pricing or regulatory filings.

How marketers should use synthetic respondents

1. Rank, don't size. Use AI to narrow three concepts to one; never put "62% purchase intent" from AI into a report.
2. Check the distribution first. If everyone is "very satisfied" and nobody is unsure, you are looking at stereotype data.
3. Ask humans about recent events. Price increases, competitor news and current memes are where twins fail most.
4. Use AI to improve human research. Pre-test questionnaires and generate hypotheses, then validate with real customers.

For a different predictive approach, see how Kroger Predicted E-Commerce Conversions With 81% Accuracy Before Launch. For AI's impact on marketing teams, read Gartner: AI Could Eliminate Entry-Level Marketing Jobs by 2030.