Political polling in Canada is at a crossroads, as new technologies push old methods of polling aside and a series of missed election calls shine a spotlight on the industry. In the second of a series of three articles looking at how polling is done in Canada, we look at one method that is becoming more common as well as controversial: interactive voice response telephone polling.
You are probably too familiar with robo-calls – those annoying automated phone calls that deliver pre-recorded notifications about cruises you have won or 'an important message about your credit card debt.' But when that technology is coupled with interactivity, you get interactive voice response (IVR), a staple method of the polling industry.
The technology is increasingly used for political polling because it can reach a large number of people quickly and cheaply. But it is not without its issues, as the methodology suffers from lower response rates than traditional telephone polls as well as a bad reputation due to its undeserved association with those pesky robo-calls.
How IVR polling is done
To conduct IVR polls, computers are programmed with a questionnaire and calls are made automatically. Respondents hear a recording and are invited to input their responses via the keypad. For example, a question about voting intentions would ask survey-takers to press 1 for Liberal, 2 for Conservative, 3 for NDP, etc. The computer tracks these inputs, making the process completely automated and very efficient in terms of both time and cost.
Proponents of the technology emphasize that it is not in and of itself a different methodology, but just a different way of reaching Canadians. It is not about how people are reached, argues Frank Graves, President of EKOS Research, in an e-mail. "The sampling strategy, call back regimens, data purification techniques, instrument design, etc. are all crucial and determine the accuracy of the survey results."
Like many polling firms, EKOS also has the ability to conduct polls with live-callers and over the Internet as well, but it is one of two national firms, along with Forum Research, to routinely use IVR technology to conduct political polling for the media.
The advantages
Despite reaching people over the telephone in a similar manner, IVR has an edge on traditional telephone polling in several ways. One is the ease of assembling a very large sample. As hundreds of automated calls can take place simultaneously – whereas live-caller polling firms are limited to the number of interviewers they have on staff – tens of thousands of calls can be made in a matter of hours. This makes it possible to build significantly larger samples in less time than with live-callers.
The most important advantage, particularly in a time of shrinking government and media budgets, is the lower cost associated with IVR polling. Depending on the size of the sample and the number of questions, a national poll can cost as little as a few thousand dollars. For some firms, the cost is so low that the polls are freely given away for the publicity in order to attract new corporate clients.
The removal of a live human at the end of the telephone line also has its benefits. It ensures that questions are asked identically every time, as opposed to a team of live interviewers who may have different accents or inflections that could have an effect on the poll's results. And respondents may be more likely to tell the truth to a heartless computer than a judgmental live interviewer.
The disadvantages
But while speed and cost are advantages IVR has over live-caller polling, they can also lead to problems. Because of how cheap it can be to do an IVR poll, the cost of admission into the industry is very low and there is an accordingly higher risk of polls being done sloppily. It thus requires a higher degree of transparency so consumers of polls can see what is being done behind the curtain.
"Pollsters should produce or have available both weighted and unweighted results," says Mr. Graves. "If weighted results look really different from the unweighted, then something is wrong with the sample."
Weighting is something every pollster does in order to get the sample more reflective of the general population. If too many men were polled, for example, their weight in the sample needs to be reduced in order to have the correct proportions. However, according to Mr. Graves, "it should be a fairly light touch...It should never be used to correct huge deficiencies."
IVR polls also tend to have lower response rates than their live-caller counterparts. An IVR poll done in a single evening can have a response rate of 2 per cent or lower. Doing a poll so quickly can be seen as an advantage – the information can be reported the next day and be as fresh as it gets – but it comes at the cost of call-backs and other measures that can boost the response rate. With a more patient approach, the response rate can be more than doubled.
Due to the automated nature of the poll, the questionnaire itself has to be much shorter to ensure that people stay on the line. "Vote intention – when not undertaking the valuable and lengthy strategic lines of inquiry – is probably an optimal length for IVR interviews," says Doug Anderson, senior vice-president of Harris-Decima.
The most important handicap facing IVR polling, however, might be how it is perceived. "The use of IVR is associated with reputational issues, particularly in Canada," says Mr. Graves. "Sloppy applications of IVR and nefarious connections to some vote suppression activities have done nothing to help this problem." The CRTC has stricter guidelines for this sort of technology, and has recently cracked down on its improper use – for example, the failure to identify who is sponsoring a call. Some Canadian politicians have been fined. This has the potential to change the way internal political polling is conducted, as respondents may act differently knowing that a poll is being paid for by a particular party.
The track record
Despite the unique issues related to IVR polling, it has performed well in recent elections. On average, IVR polling was the most accurate in the last provincial elections in Alberta and Ontario, performed better than online polls in British Columbia, and better than live-caller telephone polls in Quebec and in the 2008 federal election. The best individual poll in the elections in Ontario, Alberta, and B.C. were all done via IVR. It makes for a track record that is hard to argue against. But it has not had success everywhere: automated polls performed significantly worse than other methodologies in the 2012 U.S. presidential election. It is not clear why IVR polling has had more success in Canada.
The future of IVR
The lower cost and comparable performance of IVR polling with other methods ensures that it is likely to remain a mainstay of Canadian media polling for some time to come. But the controversy over the use of robo-calls for other purposes, and the attention the CRTC has paid to the issue, makes the future of IVR polling in Canada difficult to predict.
And the lines of how to classify IVR polls are becoming increasingly blurred. North American polling firms are starting to blend samples reached by live-callers or online with their IVR polls in order to increase coverage and mitigate each method's limitations. "The neat boundaries across the methods of contacting respondents," says Mr. Graves, "are no longer so neat."
Éric Grenier writes about politics and polls at ThreeHundredEight.com .