Bad Sample and Misleading Results: A Cautionary Tale
By Andrew Grenville, Chief Research Officer | November 27, 2018
Sampling is not something people think much about. It gets taken for granted and is often treated like the making of sausages—people don’t necessarily want to know what goes into it. But people ignore the issue of sample quality at their peril, because bad sample can lead to deeply misleading results. And when those misleading results are made public, they undermine the reputation of the entire insights industry—as we’ll see in a cautionary tale from a recent election.
Conflicting results, insults and lawsuits
Calgary, Alberta, Canada is a beautiful city of just over 1.2 million people. It is picturesquely situated at the confluence of two rivers in the foothills of the majestic Rockies. It is also a place where polling went horribly awry, in a very public way.
The incumbent mayor of Calgary, Naheed Nenshi, was running for a third term in late 2017. Mainstreet Research, headed by Quito Maggi, was doing polling for Postmedia, a company that owns two prominent newspapers: the Calgary Sun and the Calgary Herald.
The sample for the Mainstreet survey came from using interactive voice response (IVR). That’s the famous “Press one if your answer is ‘yes.’ Press 2 if your answer is ‘no.’” IVR has a very, very low response rate. Its representativeness and reliability are, therefore, highly questionable. However, because it is automated, it is very inexpensive to call tens of thousands of numbers—even if almost all of them ignore the call or hang up.
Mainstreet and Postmedia released three polls in the weeks before the election, predicting Mayor Nenshi would lose to a relatively unknown challenger named Bill Smith. These results were at odds with other companies’ surveys—which were only released later—as well as the mayor’s personal polling.
Local political science professors expressed skepticism, and the Twittersphere blew up with statements like: “Would never trust this poll . . . . it's Postmedia propaganda” and “Cons[ervatives] just trying to swing an election with rigged poles [sic]. . . . we are not impressed.”
When a poll was publicly released that contradicted their numbers, Mainstreet’s Quito Maggi tweeted, "If polling were poker, this is the part where I would go all in; I would bet $10 million we're closer than that pseudo poll today."
When political scientist Duane Bratt tweeted, “Asking Canadians just released a poll on mayor’s race that is pretty much the opposite of previously released polls by Mainstreet,” Maggi fired back with “Your credibility is getting stretched the more you comment, let’s just see in 5 days.”
When the mayor’s pollster Brian Singh publicly expressed his doubts, Mainstreet slapped him with a “cease and desist” letter threatening legal action for making libelous statements. The letter said that if the mayor’s pollster continued to make “accusations or refuse[d] to retract [his] previous statements publicly," Mainstreet would “be forced to take legal recourse for the damages inflicted."
Mainstreet executive vice-president David Valentin also came out swinging, suggesting commentators were biased. "Certainly I've seen a lot of behaviour from political scientists that I would say is quite shocking in this election campaign, and some of it, quite frankly, is quite appalling," he told 660 News. Valentin also said that the company planned on "singling people out" for "what exactly it is they said and did" about the poll results. "I think anyone who comments to the media should expect that their comments are going to receive scrutiny after the fact," he said. "I think that's fair."
The Friday before the election, Valentin tweeted, "Some people are going to have a very bad Monday, but not me."
Mainstreet predicted an 11-point lead for Smith. In the end, Naheed Nenshi was re-elected with an 8-point lead over his top competitor. Whoops.
We all get to share in the hangover
On the Mainstreet website the next day Maggi wrote, “On Monday night, I watched with utter shock and embarrassment as results came in for municipal elections across Alberta and our final prediction in the Calgary municipal election was completely and totally wrong. Our final tally showed an 11-point win for challenger Bill Smith over incumbent Naheed Nenshi. The result was a 7-point win for Nenshi. Our final poll had underestimated the incumbent’s vote by 12% and overestimated the challenger by 8% for a total deviation of over 20%.”
Maggi admitted to CBC News that the second poll his company did, which had Smith ahead by 17 points, was based on a "wonky sample." "We knew that it wasn't a great sample, but it's the sample we get," he said.
Pollster Marc Henry of ThinkHQ Public Affairs, who conducted internal polling during the race, said his research showed that Nenshi had started with a 20-point lead among decided voters, and that the gap narrowed during a contest that got nasty. If that’s the case, then Mainstreet could have been off by as much as 37% in the earlier stages of the race.
“We didn’t put this number out there to be malicious, or interfere with democracy, or anything like what we’ve been accused of,” Maggi told Global News. Mainstreet conducted a full investigation and promised to tweak their methodology.
Saskatchewan party leadership race
A few months later, Mainstreet did polling in neighboring Saskatchewan. There, the very popular Saskatchewan Party was electing a new leader to replace Brad Wall who, for many years, was the most favored premier in Canada.
Mainstreet released numbers a few days before that election. Global News reported Ken “Cheveldayoff has 46.2 per cent support among decided and leaning Saskatchewan Party voters, with Scott Moe the second choice at 21.5 per cent. They are followed by Alanna Koch (19.5 per cent), Gordon Wyant (9.7 per cent) and Tina Beaudry-Mellor (3.1 per cent).”
In the end, Scott Moe received 54% of the vote, and Cheveldayoff came third. Whoops.
The impact on public perception
Reg Downs is Senior Advisor to the Government of Saskatchewan. He has also handled his party’s polling file for its provincial election campaigns. These kinds of bad polls caused him problems because they influence public opinion. We spoke after the leadership campaign. “The problem is,” he said, “media outlets do publish these things. People do tend to read them. I don't know to what extent they influence people, but I think they create an impression this party is winning, or this party is losing, and we have seen some terribly inaccurate stuff recently.”
This puts Reg and the Party in a difficult position. “They put out some really inaccurate polls in Saskatchewan here,” said Downs, “and then you're questioning their accuracy and their methodology. When you do that as a political party, it's difficult because you can come across as whiney. ‘Oh, you are just saying that because you are losing.’ No, we are saying that because they're inaccurate and they have a history of being inaccurate. So you are always having this discussion: Do we just ignore it? Do we let it go? Do you try to counter it in some way?”
He also noted poor-quality polls undermine the reputation of research. “A few examples of bad polls can make people question the entire industry,” he said. “Even when you have what we consider to be an accurate pollster, people are sort of skeptical of the whole industry.”
I use these examples not to pick on IVR, because there are many other terrible ways to sample too, but because they dramatically demonstrate how bad methodology can bring the industry into public disrepute. In recent years, more poor-quality polls (combined with an over-enthusiastic media) have caused many to question the validity of survey research.
Following the erroneous predictions leading up to the election of Donald Trump in 2016, the New York Times reported, “It was a rough night for number crunchers. And for the faith that people in every field — business, politics, sports and academia — have increasingly placed in the power of data.”
Some very public polling misses in the U.K. provoked the House of Lords Committee on Political Polling and Digital Media to commission a report to consider the “effects of political polling and digital media on politics.” In its report, the committee stated, “Our central concern was that, if it is becoming less likely that polls can provide accurate estimates of the likely election outcomes, then there is a significant risk that future elections will be affected by misleading information, potentially distorting the democratic process.”
The report went on to say that the available data on longer-term polling performance trends suggest that “it would not be correct to say that we are witnessing a decline in the accuracy of polling” but that “although polling performance has not worsened in a statistically significant way, there is little doubt that confidence in polling has been shaken.”
This skepticism has implications far beyond political polling. CEOs, marketers and other stakeholders in the industry follow the news too. And so do the public we ask to answer our surveys. After the Calgary debacle, Canadian industry group Marketing Research Intelligence Association launched their own investigation. Their CEO said, "We don't want the Canadian public to perceive that polling is a wasted exercise. Because it isn't, it is the voice of the people, but it has to be performed properly and it has to be reported properly."
Amen.