How Large Language Models Are Revolutionizing Survey Research: A Deep Dive into Semantic Similarity Rating

The Problem with Traditional Survey Research

Consumer research has long been the backbone of product development and marketing strategy. Companies spend billions annually on surveys to understand purchase intent, brand perception, and customer satisfaction. But traditional survey methods face significant challenges:

Panel fatigue and satisficing: Respondents rush through surveys, selecting neutral options or providing inconsistent answers just to complete the task.

Positivity bias: Online panels tend to skew positive, making it difficult to distinguish between truly promising concepts and mediocre ones.

Cost and time constraints: Recruiting qualified respondents, especially for niche demographics or B2B audiences, can be expensive and time-consuming.

Limited scale: Testing dozens of product concepts or ad variations becomes prohibitively expensive with traditional panels.

These limitations have pushed researchers to explore whether artificial intelligence could supplement or enhance traditional survey methods. The question was never whether AI could generate text that sounds like survey responses. The real challenge was whether those responses could actually predict real consumer behavior.

Enter Semantic Similarity Rating

A team of researchers from PyMC Labs and Colgate-Palmolive recently published findings that represent a significant breakthrough in this space. Their methodology, called Semantic Similarity Rating (SSR), takes a fundamentally different approach to extracting survey-like data from large language models.

Why Direct Likert Ratings Fail

Previous attempts to use LLMs for survey simulation typically asked models directly for numeric ratings. "On a scale of 1 to 5, how likely would you be to purchase this product?" The results were disappointing. Models consistently produced narrow distributions clustered around neutral values, rarely outputting extreme ratings like 1 or 5.

This "regression to the mean" behavior made the synthetic data nearly useless for practical applications. While the models could sometimes rank concepts in roughly the right order, the compressed distributions meant researchers couldn't distinguish between strong and weak performers with any confidence.

The SSR Breakthrough

The SSR methodology flips the script entirely. Instead of asking for numbers, it lets the LLM do what it does best: generate natural language.

Here's how it works:

Step 1: Persona conditioning. Each synthetic respondent is given a demographic profile matching real survey participants. Age, gender, income, location, and category usage patterns are all specified in the prompt.

Step 2: Free-text elicitation. The model is asked to explain, in its own words, how likely it would be to purchase the product and why. No numeric scale is mentioned.

Step 3: Semantic mapping. The generated text is compared against carefully crafted "anchor statements" that represent each point on the Likert scale. Using embedding models, researchers calculate how semantically similar the response is to each anchor.

Step 4: Probability distribution. Rather than forcing a single rating, SSR produces a probability distribution across all scale points. A response might be 60% similar to "I would probably buy this" and 30% similar to "I might consider it."

This approach produces distributions that closely match real human survey data, with the full range of responses from enthusiastic early adopters to skeptical non-buyers.

The Numbers That Matter

The research team validated SSR against 57 real consumer surveys covering personal care products, with over 9,300 human respondents. The results were striking:

90% correlation attainment: SSR achieved approximately 90% of the correlation that human test-retest reliability would predict. In practical terms, this means synthetic rankings of product concepts matched human rankings almost as well as one group of humans would match another.

85%+ distributional similarity: Using the Kolmogorov-Smirnov statistic, SSR distributions matched human distributions with similarity scores exceeding 0.85. The synthetic data captured not just averages, but the full shape of response distributions.

Reduced positivity bias: Interestingly, synthetic respondents were actually less positively biased than human online panels. They gave harsher ratings to weaker concepts, potentially providing better signal-to-noise ratio for concept screening.

Preserved demographic patterns: The methodology successfully reproduced known demographic effects. Middle-aged respondents showed higher purchase intent than younger or older groups. Budget-conscious personas rated products lower. Category-specific preferences aligned with expectations.

Practical Applications

These findings open up several practical applications for market researchers and product teams:

Concept Screening at Scale

The most immediate application is rapid concept screening. Instead of testing 5 concepts with 400 respondents each, teams can now screen 50 concepts synthetically, identify the top performers, and then validate those finalists with human panels. This approach dramatically reduces costs while maintaining research quality where it matters most.

Early-Stage Exploration

For highly confidential or early-stage concepts, synthetic testing provides a way to gather directional feedback without exposing ideas to external panels. Product teams can iterate on positioning, pricing, and feature combinations before any human sees the concept.

Demographic Deep Dives

Synthetic respondents can be configured to match any demographic profile, making it possible to explore niche segments that would be expensive or impossible to recruit in sufficient numbers. Want to understand how left-handed vegetarian millennials in rural areas might respond to your new product? SSR makes that feasible.

Survey Design Validation

Before fielding expensive surveys, researchers can use synthetic responses to test question wording, identify confusing items, and validate survey logic. This catches problems before they affect real data collection.

Limitations and Best Practices

The researchers are careful to note that SSR is not a replacement for human research. Several limitations deserve attention:

Domain dependency: Performance depends on how well the LLM understands the product category. Highly technical or niche domains may see reduced accuracy.

Demographic fidelity: While major demographic effects are reproduced, subtle cultural or regional variations may not be captured accurately. Gender and ethnicity effects, in particular, showed weaker alignment.

Anchor optimization: The anchor statements were tuned on a specific corpus of consumer research. Applying SSR to different survey types (political polling, employee engagement, healthcare) would require developing new anchors.

Structural limitations: Synthetic respondents lack real-world constraints like actual budget limitations, prior brand experiences, or social influences that shape real purchase decisions.

The recommended approach is to use SSR as a complement to human research, not a replacement. Use synthetic data for exploration, screening, and iteration. Use human data for validation, final decisions, and tracking.

What This Means for the Future of Research

The SSR methodology represents a significant step forward in making AI useful for market research. By working with the natural strengths of language models rather than against them, researchers have found a way to extract meaningful, actionable data.

For companies like PollGPT, these findings validate the potential of AI-assisted survey research. The technology is not about replacing human insight but about augmenting it. Faster iteration, broader exploration, and more efficient use of research budgets all become possible when synthetic and human data work together.

The research community is already building on these foundations. Future work will likely focus on automatic anchor optimization, broader survey construct support, and hybrid calibration methods that combine synthetic and human data more seamlessly.

For now, the message is clear: AI-generated survey responses have moved from interesting experiment to practical tool. The question is no longer whether they work, but how best to integrate them into existing research workflows.

---

References

1. Maier, B.F., Aslak, U., Fiaschi, L., et al. (2025). "LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings." arXiv:2510.08338

2. PyMC Labs. (2025). "AI-based Customer Research." pymc-labs.com

3. GitHub Repository: Semantic Similarity Rating

---

The SSR methodology described in this article is implemented in PollGPT's AI Simulation feature, enabling researchers to generate synthetic survey responses that closely match human data distributions.