B.C. premier-elect Christy Clark pauses during a news conference at her office in Vancouver on May 15, 2013, after winning a majority in the provincial election Tuesday.Darryl Dyck/The Canadian Press
Canada's pollsters woke up Wednesday with what is becoming an eerily familiar feeling – egg on their face, so to speak, after the B.C. election results. How could they have so badly underestimated Christy Clark's late surge?
The result comes after similar ones in Quebec and Alberta: The polls didn't predict election night. Alberta Premier Alison Redford, once a polling underdog herself, couldn't help but congratulate Ms. Clark's Liberals for "proving the pollsters, pundits [and] political scientists so spectacularly wrong."
Some pollsters were rolling out mea culpas on Wednesday. "Clearly we missed on some of it pretty badly," Ekos Research Associates pollster Frank Graves said. Others said it's not that simple.
Polls have become a pillar of election campaigns – with newspapers and other media relying on them in their coverage of how a race is evolving. Pollsters say they are searching for answers about what went wrong and are refining their methodologies, and some warn their credibility is at stake. But they also say the industry is changing along with Canada, with shifting demographics and forms of communication, and the evolving nature of low-turnout, last-minute-decision campaigns.
And pollsters weren't all entirely wrong. A Forum Research poll last week did forecast a slim majority government for Ms. Clark, and many polls showed a narrowing result and a horse race emerging. Some also saw large swaths of undecided voters.
As such, not everyone thinks pollsters failed. Pollster Angus Reid called this race "polling at its finest," suggesting the NDP were, in fact, ahead and that Liberal voters were motivated by polls showing just that.
Regardless, the polling industry is one in transition, struggling to refine methods to cope with several factors.
Methodology
Not all polls are created equal. There are three general strategies: live phone interviews, automated phone polls and online panels that use pre-assembled lists of participants.
In B.C., phone polls fared much better than online polls.
The poll that best predicted Monday's results was an automated phone poll, known as "IVR," or Interactive Voice Response. Published five days before the election, Forum Research found the Liberals were just two points behind, but predicted the vote was spread out in such a way that they'd actually win 43 seats, or a one-seat majority. The Liberals ended up winning 50. "It wasn't bad. It could have been better, of course," Forum President Lorne Bozinoff said.
IVR polls are quick. However, some other pollsters dismiss IVR as too randomized and unscientific.
"It's robocalling, basically, and most pollsters wouldn't give it the time of day. I think [Forum] just happened to luck out on that one," said Steve Mossop, president of Insights West. Its most recent poll, on May 2, was done online and found the gap closing, but had the NDP still up eight per cent.
"I think there is some serious question or doubt [about pollsters' credibility]. Pollsters have always hung their hat on this – if we can predict elections, we can predict other things. This is a blow to the industry. It's not insurmountable, but it's a blow for sure," Mr. Mossop said.
Some phone polls can call cellphones and, therefore, reach people without land lines. But not all do, and Forum's did not. Live interview telephone polling also takes longer and is more costly, but many see it as more reliable.
"Even with declining response rates, it is still the best way to collect data, by far and bar none," said Paul Seccaspina, president of Oraclepoll Research, which used live interviews to publish a poll, last week, showing Ms. Clark catching up but down four points. She won by five.
Online panels did, on average, the worst. Critics say they tend to skew towards younger voters and have to be adjusted. They also say there's no proven way to estimate a margin of error. Online polls failed, said Dimitri Pantazopoulos, Ms. Clark's former principal secretary and a Liberal pollster. "I think [most pollsters] had some kind of flaw, and they called it wrong. ... The numbers were solid, they were trending in the right direction," he said.
Timing
Pollsters, in defending themselves, have a reminder: Campaigns matter.
In Alberta's election last year, most polls showed a large lead for the Wildrose Party, but they were conducted before a series of controversies led the party to stumble down the stretch. Ms. Redford won.
The late swings are due, in part, to undecided voters. Mr. Seccaspina's poll found 24 per cent of respondents hadn't decided, while Mr. Mossop found 15 per cent had not, but were leaning Liberal. An Ipsos Reid exit poll found 11 per cent of voters decided in the voting both.
"One lesson we've learned from last night is we probably shouldn't stop [polling] four or five days before the election day. And we learned that in Alberta," said Mr. Bozinoff, the Forum pollster.
Media coverage, meanwhile, feasts on any poll that comes out. Mr. Seccaspina said that's a problem – polls are a snapshot, not necessarily a predictor.
"I think part of the story here is how they're reported. And I think they've got to stop being reported that this is indicative. People change their minds," he said. "Bottom line is voters matter, elections matter and I think we really need to get away from the polls being the news story."
The work in determining what went wrong began before the results were in: Ipsos Reid did exit polling on Tuesday, interviewing 1,400 voters. Among those who decided their vote before the campaign, the NDP held a huge lead; among those who decided during the campaign, the Liberals led, the exit poll found.
Negative ads run by the Liberals "had a slaughtering effect," Ipsos Reid said, concluding: "campaigns matter, negative advertising can have a huge impact, and motivating voters right down to the wire can have a huge influence."
X-Factors
Undecided voters are one of many moving targets giving pollsters trouble. Another major challenge is voter turnout.
Preliminary figures show turnout was at 52 per cent in B.C., which was similar to 2009 levels but well below 2005 levels. At that level, it's difficult to predict who will show up.
"It depends election to election who gets their vote out, and I think the B.C. Liberals got their vote out," said Sarah Weddell, vice-president of Hill and Knowlton, which used an online poll five days before the election to find the Liberals down six points. "What happened on election day was the NDP vote did not show up," added John Wright of Ipsos Reid.
Another factor is vote-splitting. B.C. and Alberta each had four-way races. Ms. Redford benefitted from a slumping Liberal vote, while Ms. Clark benefitted from a slumping Conservative vote. The B.C. NDP, meanwhile, lost ground to the Green Party.
Some first-generation Canadians are also difficult to poll, and in some cases prefer to be polled in their mother tongues, Mr. Reid said. That left question marks in B.C. regions with heavy visible-minority populations.
And in B.C., Alberta and Quebec, incumbent parties did better than expected, Mr. Graves noted.
"All three came back in the late stages and registered 10 points ahead of where they were in the polls. ... So, obviously, there is something going on," he said. Ekos's phone poll, on the eve of the election, found the NDP up five points. "It gets discouraging, frankly. You're doing this to try and hone your skills. And when it doesn't work, it's no fun."
What now?
It's back to the drawing board for some pollsters – though not all. Mr. Wright, for instance, says Ipsos is satisfied with its methodology, saying voters "upset their own apple cart based on everything they've seen, read or heard."
Some pollsters are juggling how to blend polling models. Online panels are used widely in consumer research, and tend to reach young people. Phone calls, meanwhile, tend to target older people and face a declining response rate – people aren't as likely these days to talk honestly to a pollster.
"We're evolving to [online] methods that, in a way, favour the younger voter, and the younger voter is not a traditional voter … the election process favours older people," pollster Barb Justason said. Justason Market Intelligence's online poll had the NDP up 14 points on May 10, only to see them lose four days later. The industry has a methodology problem, she said.
"We're dealing in a world where we cannot produce a probability sample, no matter how hard we try, hold our breath and spin around three times. We can't do it," she said.
Abacus Data said it used the same methodology to much stronger results in Ontario's election, but will review it. "I think there's a number of things we need to consider. It's certainly a wake-up call for a lot of us," CEO David Coletto said. Abacus's poll, early in the campaign, found the Liberals down 10 points.
Calgary pollster Bruce Cameron said the industry needs a mix of social-media monitoring, online responses and live telephone polling to get the best picture.
"It's not about going where the puck is. It's about anticipating where the puck will be," said Mr. Cameron, who accurately predicted Ms. Redford would win and didn't poll in the B.C. race. "It is an amazing [Clark] comeback, but the seeds of it were there."