Everyone says synthetic research is a complement. Most validation evidence tests it as a replacement.

By Elizabeth van Monsjou, Senior Director of Venture Strategy, Research, Highline Beta

Matching human answers is the right test for a tool that replaces fieldwork. A tool that complements fieldwork needs different tests.

Ask most synthetic research vendors if their tool replaces talking to real people, and they'll tell you no. It's a complement [1]. We say the same thing about Candor [2].

Then ask how they prove their tool works. Most will show you how closely their synthetic answers match a real human sample [1][3], and therefore how well the tool stands in for human research. This site’s industry guide calls that the key question [4].

But that doesn’t answer the question of how a synthetic tool intended to complement human research and make it better and cheaper proves its value. Four key answers any synthetic research tool should be able to answer:

• Are participants consistent within themselves and factually accurate?

• Are the participants it generates distinct from one another?

• Is it just falsely validating a customer’s preconceived notions?

• Are its participants traceable?

Matching still matters in the right context. A recent article on this site laid out how to score a synthetic survey properly [3]: show what a random guess would score, show the best score realistically possible, and show where the tool lands between the two. For a tool that claims to replace fieldwork, that's the standard. Hold vendors to it.

Where matching is the right test

Matching human data is the right bar in two cases. The first is when a decision depends on a number about a population (how many people will buy, which concept wins). The second is when a vendor sells synthetic research as a stand-in for recruited respondents.

In either case, ask three things of the evidence:

• What would a blind guess have scored? A match only means something next to a baseline. Without one, you can't tell an impressive result from a lucky one.

• Was the tool tested on human data it had never seen? If it already saw the answers, a match only proves it remembered them. Prediction means getting answers right that it didn't have.

• Where did it miss? A good overall score can hide misses that keep landing on the same groups of people or the same types of questions. Those patterns tell you where not to trust it.

The AI model itself can also remember. If a vendor tests against a famous public survey, the results may already be in the data the model was trained on [5]. Ask vendors for tests on results published after the model was trained, or on private data that was never online.

Candor doesn't claim to replace fieldwork. So the rest of this article is about how to test synthetic research as a complement to human research.

What the complement claim commits to

A Candor participant is built from what's already been documented: industry reports, published research, professional association data, social media posts and documents you upload. That also sets its limit. Candor is strong on what’s already known and weak on what nobody has written down yet. For that, you still need real people.

What it can do is show you things that are new to you, quickly, in a form you can interview. Think of it as secondary research that answers follow-up questions.

Our rule for when to trust it comes down to two questions. Will the market (and real customers) tell you quickly if you got it wrong? And can you change course when they do?

If the answer to both is yes, synthetic findings are enough to act on. Which message to send, which value proposition to lead with, which feature to explore first. If the synthetic read was wrong, reply rates and usage show it within weeks. There's still a human check. It just comes from live customers instead of a recruited panel.

If a decision is hard to undo, don’t make the call on synthetic findings alone. Use them to narrow the options, then confirm with real people. Synthetic research should never kill a concept on its own, approve a build, name a product or set a price. Those decisions should be made by real people.

That changes what testing has to show. A synthetic participant needs to:

• Be right about the world it describes. Confident claims have to hold up against the facts.

• Be different from the other participants. Otherwise there's no point interviewing more than one.

• Be willing to disagree with you. A participant that folds under pressure just reflects your assumptions.

• Be traceable to its evidence. You should be able to see what it was built from and check it.

Each of those can be tested. Here's how we test them in Candor, including where we still have work to do.

Test 1: Does it state false things confidently?

A synthetic transcript reads the same whether its claims come from evidence or were made up. So ask two questions. What keeps a participant consistent with itself? And what checks its claims against the real world?

In Candor, a second AI model (we call it the critic) checks every participant’s answer before you see it. It compares the answer against what the participant has said they believe, what they've recently decided, and the conversation so far. If it finds a contradiction, it rewrites the answer, up to twice.

In one of our validation tests, a synthetic user built an entire interview around a confident claim that got the basics of a sensor technology backwards. Every answer was specific and consistent with the last. The critic had nothing to catch, because the participant never contradicted itself. It was just wrong.

So we're building a fact-checking step. It's tricky. Real people are confidently wrong too, so the check has to judge whether an error actually affects the research or is just the kind of mistake a real person would make?

Our test: in two interviews we ran by hand, we told synthetic participants false versions of what they'd said earlier or what was in their profile. Later, we asked about small facts again in different words. Neither participant clearly contradicted themselves. One answer was borderline, and a second rater who didn't know the initial rating judged it compatible.

That matches how the critic works: it rewrites clear contradictions and lets borderline ones through. Because rewrites don't show on screen, a test like this only catches what gets past the critic. You can run the same test yourself in Candor's manual interview mode.

Test 2: Are twelve participants actually twelve people?

Research on synthetic respondents keeps finding the same problem: their responses are less varied than real people’s [6][7][8]. Answers bunch up around the average, and the unusual views at the edges go missing.

The quick check is to ask a vendor for two transcripts from the same study, side by side. If they read like one voice with two names, you have a focus group of one.

Human samples bunch up too, but almost nobody measures it. With a synthetic audience, it's easy to measure. Ask your vendor if they do.

Our first test: we took 22 interviews from one audience, split across six participant groups, and removed the names. Two AI models from different providers compared the transcripts in pairs, mostly within the same audience groups. They checked whether two transcripts read like the same person, told the same story or used the same specific details. Shared opinions and themes didn't count.

Neither model found two participants who read as the same person, and no story repeated across the study. One model did find some overlap in five of the 31 pairs within groups, but that’s to be expected since groupings are constructed based on similarity.

Our second test, against real people: we ran one concept-testing study with 12 real participants (from private client research that was never online) and with 57 Candor participants across three audiences.

Candor found most of the main themes real participants raised, but it missed about one in five. When we compared two groups of real people, they missed about one in seventeen of each other's themes. Candor's groups also repeated each other's themes more, and they missed the unusual views a few real people raised.

Concept-testing study	Candor	Real people
Participants	57, across three audiences	12
Main themes a group of six found	75 to 80%	94%
Main themes a group of six missed	About 1 in 5	About 1 in 17

That's the same narrowing the research describes. It's one study, and it's the problem we're working on in Candor right now.

To check this yourself, pick two participants from the same group, cover the names and read their interviews side by side. Do they sound like the same person, tell the same story, or provide the same details?

Test 3: Does it tell you what you want to hear?

AI models tend to agree with whoever is using them [9]. Real participants do this too (every qualitative researcher deals with it). But in AI models, the habit comes from how they're trained.

That matters when looking for usable feedback. A synthetic participant that gives in under pressure just hands your own assumptions back to you as findings. Being "grounded in a profile" describes how a participant was built. It says nothing about how it acts when you push.

Our test: we picked eight participants, four at random and four chosen to cover a range of personalities (e.g., low versus high agreeableness). For each one, we took two positions they'd stated clearly in an earlier interview. We pushed on one with leading questions, and on the other by telling them we thought they were wrong. Each position got three rounds of pressure with no new reasons, only social pressure.

None of the eight dropped a position. Three or four (depending on the rater) gave some ground while keeping their view, which is what a thoughtful person does. A second AI model from a different provider, which didn't see our scores, scored all 48 answers and agreed with the original rater on 41.

8 participants, 48 pushed answers	First rater	Second rater
Answers that dropped the position	0	0
Answers that gave some ground	8	5
Answers that held the position	40	42
Answers scored unclear	0	1
Participants who gave some ground	3 of 8	4 of 8

To check this yourself, push back on a position a participant has stated, three times, without giving a reason. Watch whether they keep it, soften it or drop it. If they drop it, ask yourself whether you gave them a reason to or if they’re just giving in.

Test 4: Can you see the background evidence?

If a participant is built from evidence, you should be able to see that evidence and check it yourself. Not take the audience on faith.

In Candor, every evidence-based field on a participant's profile carries one of three labels:

• Grounded: it comes from a source you can check.

• Derived: it follows from the evidence, but no single source says it.

• Assumed: no evidence covered it, so the AI filled the gap.

Demographics, personality scores and bias scores don't carry labels. They come from population data and from the ranges the evidence sets for each group. Links to the sources live on the audience page, where each piece of evidence opens to its original page.

Our test: we clicked through one audience built from 378 pieces of evidence. All but one linked to a working page, and that one was labeled Derived (which is what the label is for). The eight pages the audience relied on most were all real, though two asked for an email address first and one was a 2018 report copied onto another site. On the participant we checked, Candor had verified 22 of the 25 Grounded fields that cite a source.

To check this yourself, find the source an audience leans on most and read three pieces of evidence taken from it. Then open the source and see whether it actually says those things. The label tells you where a fact came from. Only reading the source tells you whether it's true.

A complement, for now

Today we’re calling synthetic research a complement because of where the industry is, not as a matter of principle. It's new, and there hasn't been enough time to check its predictions against what actually happens with real customers. The next ten years of testing (ours included) will decide whether it stays a complement.

Synthetic and human samples go wrong in different ways. It's entirely possible synthetic data will one day predict consumer behavior and decisions more reliably than a human sample does. If the evidence shows that, our view will change.

Until then, start with one question for any vendor: what does this tool claim to do, and does its testing actually check that claim?

We'd rather you ask us these questions than take our word for it.

Sources

1. Qualtrics (2026). "Synthetic data for market research: Your questions, answered." qualtrics.com/articles/strategy-research/synthetic-data-market-research

2. Highline Beta (2026). "Introducing Candor, Highline Beta's synthetic user research platform." highlinebeta.com/blog/introducing-candor-highline-betas-synthetic-user-research-platform

3. Doudkin, A. (2026). "What a synthetic survey replication score is measured against." Synthetic Market Research. syntheticmarketresearch.com/insights/what-a-synthetic-survey-replication-score-is-measured-against

4. Synthetic Market Research (2026). "Synthetic Market Research 101: State of Industry." syntheticmarketresearch.com/insights/synthetic-market-research-compendium

5. Ludwig, J., Mullainathan, S., and Rambachan, A. (2026). "Large Language Models: An Applied Econometric Framework." Annual Review of Economics. Also NBER Working Paper 33344. nber.org/papers/w33344

6. Bisbee, J., et al. (2024). "Synthetic Replacements for Human Survey Data? The Perils of Large Language Models." Political Analysis, 32, 401-416. doi.org/10.1017/pan.2024.5

7. Wang, A., Morgenstern, J., and Dickerson, J. P. (2025). "Large language models that replace human participants can harmfully misportray and flatten identity groups." arxiv.org/abs/2402.01908

8. Banawa, R. (2026). "Synthetic Users vs. Human Respondents: Same Conclusion, Less Variation." Nielsen Norman Group. nngroup.com/articles/synthetic-users-vs-human-respondents

9. Sharma, M., et al. (2024). "Towards Understanding Sycophancy in Language Models." International Conference on Learning Representations (ICLR 2024). arxiv.org/abs/2310.13548