Buyer guide
What to ask in a synthetic research demo
Bring a real question. Ask where the answers come from. Find out what the full job will cost.
Can we try our own example?
Pick a small job your team actually needs to do. Perhaps you want to compare three messages for first-time buyers. Tell the provider which country and language matter. A demo using your question will tell you more than a polished example chosen by the sales team.
What data is behind this audience?
Ask where the data came from, when it was collected, and which people it covers. Then ask what happens to any customer data you upload. Will it be kept separate? Can you delete it? ESOMAR's questions for buyers of AI services are a useful checklist for this conversation. [1]
What does your accuracy score mean?
If a provider says a model is '90% accurate', ask: accurate at what? Picking the same winning idea as a human survey is different from predicting how many people will buy. Ask to see results for customers like yours, including examples where the model got the answer wrong.
Can you show us a fair test?
Use a study whose answers were kept out of the model's training and setup. That is often called a holdout test. Ask the provider to make its predictions before you reveal the real results. Then check the parts that matter to your decision, not just one score for the whole study.
What will the complete job cost?
Ask about setup, data preparation, usage limits, exports and support. Check that the tool handles your question types and languages; Qualtrics, for example, documents which survey features its Synthetic Panels support. Leave with a written quote for your actual task and an agreed test of success. [2]
