A spat over polling erupted last week, as Darrell Bricker and John Wright of Ipsos-Reid sent an open letter to Ontario's journalists suggesting they take more care in choosing the polls that get reported and how those polls get reported. Specifically, online and IVR polling were highlighted as deserving more questioning glances from the province's media. Is this a fair criticism?
Political polling is generally done in three ways. The traditional method is to conduct surveys over the telephone, using live operators who follow scripts in order to ensure that each operator is asking the same questions in the same order.
Another method is over the Internet, via online panels of often 100,000 people or more. As with telephone surveys, respondents are weighted by their demographic profiles in order to have a sample that approximates the general population.
A last method is via IVR (interactive voice response), often derisively called "robo-polling." These polls are also conducted over the telephone, but there is no live operator. Respondents listen to a recorded message and are asked to answer questions using their keypad (for example, "Press 1 for Conservative, 2 for Liberal," and so on).
In the last week of the 2011 federal election campaign, eight firms released polling data after having been active throughout the campaign. Three firms conducted their polls by telephone (Nanos Research, Harris-Decima, and Ipsos-Reid), three used online panels (Angus-Reid, Abacus Data, and Léger Marketing), and two used an IVR system (Forum Research and EKOS Research). Which method performed best?
Going by average error, telephone surveys were the most accurate with an average error of 1.2 points per party (Conservatives, NDP, Liberals, Bloc Québécois, and Greens). Online surveys were next best with an average error of 1.4 points (based on average error, the closest poll was conducted online), while IVR polling was off by an average of two points per party.
Three polling firms (two telephone and one internet) pegged national support for each party within their polls' stated margin of error, a feat that was far more difficult to achieve at the regional level.
In terms of party support, both telephone and online polls performed similarly, though the average of telephone polls was closer to the actual results for all four national parties than either online or IVR polling.
But the missive from Ipsos-Reid was aimed at Ontario's journalists, and the story of polling the 2011 federal campaign in Ontario is quite different. Pollsters had a great deal of trouble with Ontario, far more than in Quebec where the average error per party was significantly lower.
In Ontario, IVR and online polls actually performed best, with an average error per party of 2.4 points each compared to an average of 3.3 points for the telephone polls. Moreover, of the eight polling firms the two worst in Ontario did their polling by telephone.
Telephone polls were, however, the closest to accurately predicting Conservative support. But they still underestimated Tory support in Ontario by an average of 4.3 points, only slightly better than the online (4.4 points) and IVR (4.6 points) polls.
But IVR polling was closest to the mark for the Liberals, while the New Democratic result in Ontario was most accurately predicted by online polls.
During the 2007 Ontario election campaign, six pollsters were active in the final week. Five of them conducted their polls over the telephone, and they averaged only 1.7 points of error per party. That compared favourably to an average error of two points per party for the lone online poll. But that online poll out-performed two of the five telephone surveys.
Telephone polls are still considered the standard in the industry, and nationally they performed best in the 2011 federal campaign. Online polls are still relatively new but have proven themselves to be reliable in the right hands, while IVR polls struggled at the national level in the federal election but did better in Ontario. Which method of polling will perform best in this campaign remains to be seen, but the two best firms in the 2007 race, both with an average error per party of less than one point, did their surveys the old fashioned way.
Éric Grenier writes about politics and polls at ThreeHundredEight.com