Assessing the Reliability and Validity of Panels Around the World
By Rob Berger, Managing Director, Maru/Blue | March 17, 2020
In the world of sample, reliability and validity are the difference between research that informs and surveys that mislead. We measured the test-retest reliability and criterion validity of 28 panels in 14 countries: Australia, Brazil, China, France, Germany, India, Italy, Japan, Mexico, Russia, Singapore, South Africa, Spain and Thailand.
The result is a 14-nation whirlwind tour of how research can go horribly awry and how it can be refreshingly right. The lessons learned reveal a great deal about the nature of sample in a wide range of markets around the world.
We undertook this research as part of our Maru/Blue Sample Certified program. Our panels in the US, UK and Canada are constantly monitored for reliability and validity. Outside of these countries we rely on other sample providers, and they must be reliable and representative too. To ensure these suppliers meet these criteria, we built a global quality control program: Sample Certified.
Sample Certified tests and certifies sample vendors in countries where there is high demand for multi-market insights. Vendors who are Sample Certified become preferred suppliers.
Our objective was to identify quality panels that we can count on to deliver accurate information. We measured test-retest reliability and criterion validity. In terms of reliability, we wanted to answer the question: “If you measure the same thing twice do you get the same results?” Regarding validity, we wanted to know: “Do these results match what is known to be true?”
This systematic approach to assessing panel quality identified some great sources, some that were slightly off the mark, and some that should never be used. Validity was, overall, harder to achieve than reliability. There were some panels that returned the same results both waves but were skewed in terms of how well they represented the population. And then were a few that were frighteningly neither reliable nor representative.
There were, of course, also quite a few panels that were both valid and reliable. In total, there were 16 suppliers in 9 countries that got a score of between 85% and 100% on both reliability and validity. We call this boringly reliable. It’s not exciting, but it does allow us to be confident that information gathered from these sources will illuminate rather than mislead. So, they are boring, but in a good way.
In addition to our communities in the US, UK and Canada, this first round of research allows us to certify panels in Australia, Brazil, China, France, Germany, Italy, Japan, Spain, and Thailand.
To learn more about how sample can go wonderfully right and horribly wrong download our free whitepaper Can we count on you? Assessing the reliability and validity of panels around the world or contact us.
For an even further understanding of journey to identify quality panels around the world, watch our webinar here.