Political polling in Canada is at a crossroads, as new technologies push old methods of polling aside and a series of missed election calls shine a spotlight on the industry. In the first of a series of three articles looking at how polling is done in Canada, we look at the oldest of the methods still in wide use: live-caller telephone polling.
The odds are that, at some point in your life, a polling firm has tried to contact you over the telephone to take part in a survey. The odds are even better that the attempt failed. You were not home, you did not take the call, or you just hung up. This is what happens with roughly nine out of every 10 phone calls a polling firm makes.
And yet, in this time of decreasing response rates, the pressure on polling firms could not be any higher. Polls need to be done more cheaply and more quickly than ever before, and in the Information Age the microscope is as close as it gets. The pay-off is slim – literally, as well as figuratively. The results of a poll are discarded as soon as the next numbers are out or they are subsumed into some aggregator's formula. If the pollster is wrong, they are thrown under the bus by everyone – including the media outlet that could not afford to pay for a proper poll in the first place.
Telephone polling is in a particularly difficult position in this new reality. Though still widely considered the gold standard of public opinion polling, these surveys are becoming rarer and rarer on the political landscape. They take too long to conduct and cost too much money, and media outlets are asking themselves if that is worth the higher quality that experts assign to them.
At one point, telephone polling was the controversial new kid on the block. Polling used to be conducted door-to-door or through the mail, but once virtually every household had a telephone line it became the staple mode of contact for the industry. Today, it is the preferred method for political polling by national firms such as Harris-Decima and the Environics Research Group, as well as regional firms like Probe Research, CROP (during elections), and the Corporate Research Associates (CRA).
On the face of it, the process is rather simple. Banks of telephone numbers can be acquired from specialized sampling firms, and numbers can also be dialed randomly. Live telephone operators then follow a script and input responses into a computer (this method is often referred to as CATI, for computer-assisted telephone interviewing). If the person who is called does not pick up the phone, the number is redialed at a later time or date. Otherwise, the sample would not be as random – it would just be a sample of people who were home at a particular time.
Why they remain the gold standard
The greatest advantage of conducting surveys over the telephone with live-callers is the ability to build a random, representative sample. Being able to draw responses from a random sample is the basic necessity of any poll. Virtually everyone in Canada can be reached by telephone, and if both landlines and cell phones are included in the survey everyone in the country theoretically has an equal chance of being asked to participate. That makes the sample random.
"The methodology has stood the test of time," says Don Mills, Chairman and CEO of CRA, in an e-mail. "Its ability to produce random representative samples is still its biggest strength."
Doug Anderson, senior vice-president of Harris-Decima (a firm that also uses online and interactive voice response technology, otherwise known as robo-dialing, in addition to CATI), agrees: "the telephone remains the line of communication most able to provide the highest probability that any Canadian resident can be invited to participate." And response rates with CATI tend to be higher than with IVR polling. "It may be the case that, since response rates are lower, there are certain segments of the population willing to conduct a survey with a live interviewer, but self-exclude when it is a computer talking to them."
And why they are becoming rarer
But with response rates having dropped so low in general, there are some doubts about whether these samples can truly be considered random anymore. This criticism is often emphasized by the firms that have adopted online technologies, who themselves have come under criticism concerning the randomness of their own samples. The debate is a heated one in the industry. But the fact remains that getting people to pick up the phone is a problem.
"In some segments – especially young voters – land lines are as archaic as the rotary dial to an earlier generation," writes Angus Reid, executive chairman of Vision Critical and Angus Reid Public Opinion, in a recent article for Maclean's. "This means pollsters have a harder time finding younger voters, who either don't have a landline at all, or are loathe to answer calls from pollsters on their mobile, when they are being charged by the minute."
These limitations are not a problem for everyone, however. Mr. Mills says that a high proportion of those younger voters still live at home and can be reached by their parents' landlines. And a cynic might point out that younger people who actually vote are rare to begin with.
In addition, as polling firms adapt to include new technologies into their polling, they are also keeping abreast of how to improve older methods. "We are constantly monitoring the evolution of the telephone market," says Mr. Anderson.
But the biggest limitation facing traditional telephone polling is the cost. Staffs of interviewers need to be maintained and trained, whereas IVR polling and online panels do not have this problem. Turnover means constantly needing to retrain new hires in order to maintain the quality of polling, and interviewers need to be supervised to ensure they are following their scripts correctly.
In addition to these high structural costs, the finite number of interviewers limits the number of questionnaires that can be filled out in a given time. Responses via an online panel or IVR can be accumulated in a day or two. This is not possible with live-callers without the cost being prohibitive.
Recently, in documents acquired via access-to-information legislation, it was reported that the federal government paid $29,000 for a telephone poll of about 2,000 Canadians. A poll with a smaller sample size and only a few questions can be had for much less than that – perhaps $5,000 to $10,000 – but that is still significantly more costly than the price of a comparable IVR or online poll. As a result, "political polling is much more superficial than it was in the past," says Mr. Mills. Polls need to be kept short to keep costs down. And that means delving deeply into what is behind voters' motivations is not possible.
Their recent track record
The performance of telephone polling in recent Canadian elections has been mixed, perhaps reflecting some of this superficiality. In the few instances where the three major modes of contact were all employed, telephone polling only performed best – on average – in British Columbia's election (and in that case, it was the best of a bad lot). It had the worst average performance in the most recent provincial elections in Quebec and Ontario, as well as in the 2008 federal election. In the 2011 federal vote it was beat by online polls.
But telephone polling has had some success in recent lower-profile elections. It performed better than online polling in the last provincial elections in Nova Scotia, Prince Edward Island, and Newfoundland and Labrador, for example.
And these individual performances may not tell us much about the quality of CATI polling. Voters can change their minds between being polled and voting day, and there is no way to assess whether a pollster is accurately recording the voting intentions of the entire population, including the non-voting portion. This group can be reluctant to admit they do not plan to vote, especially to a real person on the telephone. Measuring the difference between the average person and the average voter is one of the biggest challenges facing all pollsters, and not just those who conduct their surveys over the telephone.
The problem of how to get more people to participate also needs to be tackled in order to get response rates up to where they used to be, which would help decreases costs and increase accuracy. According to Mr. Mills, "there needs to be some industry effort to promote the value of participating in survey research as a benefit for citizens as well. In a free society, how much value is associated with the ability to express your opinions in the hope of changing life for the better?"
Éric Grenier writes about politics and polls at ThreeHundredEight.com .