Buyer guide
What to ask in a synthetic research demo
Bring a real question. Ask where the answers come from. Find out what the full job will cost.
Can we try our own example?
Pick a small job your team actually needs to do. Perhaps you want to compare three messages for first-time buyers. Tell the provider which country and language matter. A demo using your question will tell you more than a polished example chosen by the sales team.
What data is behind this audience?
Ask where the data came from, when it was collected, and which people it covers. Then ask what happens to any customer data you upload. Will it be kept separate? Can you delete it? ESOMAR's questions for buyers of AI services are a useful checklist for this conversation. [1]
What does your accuracy score mean?
If a provider says a model is '90% accurate', ask: accurate at what? Picking the same winning idea as a human survey is different from predicting how many people will buy. Ask to see results for customers like yours, including examples where the model got the answer wrong.
Can you show us a fair test?
Use a study whose answers were kept out of the model's training and setup. That is often called a holdout test. Ask the provider to make its predictions before you reveal the real results. Then check the parts that matter to your decision, not just one score for the whole study.
What will the complete job cost?
Ask about setup, data preparation, usage limits, exports and support. Check that the tool handles your question types and languages; Qualtrics, for example, documents which survey features its Synthetic Panels support. Leave with a written quote for your actual task and an agreed test of success. [2]
What is included in the delivered output?
Ask for a sample export rather than judging only the dashboard. Check whether it includes the question wording, response options, audience definition, respondent-level records, weighting, generated quotations and source context. A table of answer shares and an interview transcript are different deliverables. Confirm whether your team can reproduce the key calculation outside the platform, and whether follow-up studies use the same audience or generate a new one. [1]
How do you report a weak result?
A demo should include a task where the model is less reliable. Ask what triggers an uncertainty warning and what the warning means for the proposed decision. If the provider offers a confidence score, find out whether it has been evaluated against observed errors on held-out questions. A confidence label generated by the same model that produced the response is not, by itself, evidence that the response is right. [1]
Which costs and obligations sit outside the subscription?
Separate platform access from model setup, custom audience creation, human fieldwork, integrations, support and usage charges. Find out whether credits count a study, a question or an individual generated answer. Confirm the export and retention terms for the research materials your organization supplies. Record these items with the quoted price so two offers covering different amounts of work are not compared as though they were the same purchase. [1]
What should we leave the demo with?
Keep the same brief for every supplier and leave with the proposed audience, inputs, output, evaluation design, delivery timing and commercial scope in writing. Note which capabilities were actually demonstrated and which were described as planned. Then run a bounded pilot against a relevant human study. The acceptance decision should use the original research question and success measure, not a measure chosen after the model’s strongest result appears.
