Buyer Beware!
By Andrew Grenville Maru/Matchbox | November 2022
The reliability and validity of 28 panels in 14 countries.
Sample quality is critical, especially in multi-country research. Bad sample provides misleading results, defeating the purpose of doing research. Poor quality sample adds insult to injury by dealing a blow to the reputation of our industry.
Identifying good sample from bad will make or break your project. In this paper, we convey important findings from a 28-panel, 14-country reliability, and validity, involving over 28,000 people. We report on lessons learned and propose critical questions to ask potential suppliers.
It is well known that some sample sources are better than others and that, in some countries, it is harder to get representative samples. But where is the evidence beyond anecdotal experience? What should guide buying decisions? The reputation of the supplier? Price? Using a consistent supplier across multiple markets? All online samples cover basically the same population, right?
To answer these questions and more, we conducted a study of 28 panels in 14 countries. We wanted to know if the panels were reliable, so we ran two waves and assessed test-retest reliability. We also needed to understand if these panels were truly representative of the nations they covered. So, we assessed criterion validity, comparing survey results to local census data, government household surveys, and other reliable data sources.
The nations surveyed were Australia, Brazil, China, France, Germany, India, Italy, Japan, Mexico, Russia, Singapore, Spain, South Africa, and Thailand. In each country, we obtained sample from two suppliers presumed to be quality providers based on reputation and prior experience. Each survey had samples of 500 or more people from each panel. We repeated the same study a week or more after the first wave. In countries with very high internet populations, we set quotas to ensure the population was demographically representative of the population. In countries with relatively low internet population, we set quotas to ensure the representativeness of those with internet access. The survey was mobile-friendly.
From this study, we learned several important lessons about what to look for and—more importantly—what to watch out for when sourcing multi-country samples. We propose a checklist of questions to ask suppliers when sourcing multi-country samples because the valuable insights our organizations need can only be derived from a valid and reliable sample.
To learn more about the reliability and validity of panels around the world, read our whitepaper.
Download the whitepaper
Can we count on you?
Assessing the reliability and validity of panels around the world