Synthetic user

Language models can produce interview answers, survey responses and usability feedback in the voice of any persona. That is useful for drafting questions or rehearsing a study. The pattern begins when the simulation replaces the study.

Simulated participants tend to be agreeable, articulate and average. They report intentions, not behaviour. They flatten the groups they imitate into the views an outsider would expect those groups to hold, and the flattening survives attempts to steer them: misalignment with real demographic groups ‘persists even after explicitly steering the LMs towards particular groups’. Their answers cluster: a simulated survey can match a real one on average while showing ‘less variation in responses than in the real surveys’, which makes every finding look cleaner than it is. The people most likely to be harmed, those at the edges of a group or in unusual circumstances, are exactly the people a simulation misses. The product then passes research that nobody took part in, and the team believes it has heard from its users.

The A/B test fares no better than the interview. Because models learn from observational data, changing a condition in a synthetic experiment shifts the simulated users along with it; such experiments, as one 2026 analysis puts it, ‘are more similar to observational studies than randomized experiments’.

In studies with 3,200 participants across 16 demographic identities, Wang, Morgenstern and Dickerson found that four language models standing in for human participants misportrayed and flattened identity groups. The fixes they tested ‘reduce—but do not remove—these harms’. Nielsen Norman Group compared synthetic users with real ones and found the synthetic users gave ‘shallow or overly favorable feedback’, and claimed to have finished courses that real participants had not. It concludes that synthetic users cannot produce behavioural data and should ‘complement, not replace, real research’. A 2026 review of a dozen controlled comparisons tallied more discouraging findings than encouraging ones and called the results ‘at best directional’.

The best case for simulation makes the point from the other side. Agents built by Park and colleagues from two-hour interviews with 1,052 Americans reproduced those people’s survey answers at 86% of their own test–retest consistency. The result rests on each simulated person having a real one behind them, interviewed at length. That is not research with nobody in it; it is research with people in it, reused. The pattern is the version that skips the interview.