Chrystia Freeland speaks before voting took place on Sept. 15, 2013.Michelle Siu/The Globe and Mail
Polls for the four upcoming federal by-elections on Nov. 25 suggest that the Liberals will hold on to their two seats and pick up a third from the Conservatives in formerly hostile territory. But just how reliable are these by-election polls?
The only firm brave enough to venture into the world of by-election polling in the last few years has been Forum Research, and this campaign is no different. The firm has been active in all federal and most major provincial by-elections over the last two years, and has a generally decent record. But in the most recent set of five Ontario by-elections this past summer, the winner was accurately identified in only three of the ridings, and in one of those the expected vote tallies were off the mark by a significant degree.
That does not make the accuracy of the by-election polls a coin-flip, but it does encourage a greater use of caution in analyzing the results. In fact, we can take something from the polling numbers with a reasonable degree of confidence.
The latest polls by Forum Research for the by-elections in Montreal's Bourassa, Toronto Centre and Manitoba's Brandon-Souris and Provencher were conducted on Nov. 14, interviewing between 261 and 779 decided voters in each riding. Because of those sample sizes, the reported margin of error varied between 3 and 6 per cent.
Margin of error
In practice, the margin of error in by-election polling is considerably higher. Whether it is due to the greater difficulties of building a representative sample within a single riding, the lack of interest in the campaign, low turnout, or simple methodological inaccuracy, by-election results fall within the margin of error much less often than the reported 19 times out of 20. In fact, a party's results have fallen outside of the reported margin of error roughly one-third of the time in recent by-elections.
And sample size is not always a determinant factor. Though the Forum poll with the largest sample size in the Ontario by-elections turned out to be closest to the mark, the one with the highest degree of error had a sample size larger than all but one of the federal by-election polls released last week.
This suggests that we need to apply a much larger margin of error than the sample sizes would normally demand. Based on the recent degree of error in by-election polling, we need to apply a margin of error of about +/- 10 points to the latest set of polls to achieve the same 95 per cent (19 times out of 20) confidence level. We can also apply a more useful +/- six-point band, but at only 2:1 confidence (67 per cent).
When we apply these confidence intervals, we get a less precise but probably more accurate picture of what to expect for Nov. 25's by-elections.
Who will win?
The incumbent parties in Bourassa and Provencher appear unassailable. In Bourassa, Emmanuel Dubourg of the Liberals still has a cushion of nine points over Stéphane Moraille of the NDP at the 95 per cent confidence interval. Indeed, in no recent case have by-election polls been wrong enough to erase the 29-point lead that Mr. Dubourg enjoyed over Ms. Moraille in the latest survey. In Provencher, there is the outside chance of a close result between Ted Falk of the Conservatives and Terry Hayward of the Liberals at the 95 per cent confidence interval, but the odds are heavily in Mr. Falk's favour.
Toronto Centre is less of a certainty. Though Chrystia Freeland of the Liberals sits comfortably ahead of Linda McQuaig of the New Democrats at the 67 per cent confidence interval, their support bands do overlap at the 95 per cent confidence interval. And with 11 days separating the last Forum poll from the vote, there is still plenty of scope for the NDP to take the riding from the Liberals. Nevertheless, Ms. Freeland is favoured.
The polls peg Brandon-Souris to be the closest race, with the most recent survey putting the gap at eight points. At both the 67 and 95 per cent confidence intervals, either Liberal candidate Rolf Dinsdale or Conservative candidate Larry Maguire can pull off a victory. The polls have suggested throughout the campaign that Mr. Dinsdale may manage the upset as he has been steadily leading, but the riding's fundamentals lean heavily towards the Conservatives (the Liberals took just 5 per cent of the vote in 2011, and the Conservatives or their predecessor parties have taken the majority of the vote in the riding in virtually every election since the Second World War). In addition to the extraordinary circumstances of the race in Brandon-Souris, by any objective measure this remains the by-election to watch on Monday night.
In taking into consideration these likely ranges for each of the parties, the by-election polls can be used to help gauge the race with more confidence. The Conservatives and Liberals will probably retain at least one of their seats, but the fate of the other two is less certain. The Liberals could win both of them and have a very good night, the Conservatives and New Democrats could each take one of them and grab the bragging rights themselves, or the Liberals could prevail in only one of the toss-ups. But with expectations now so high for the Liberal Party, they seem to have the most to lose.
Éric Grenier writes about politics and polls at ThreeHundredEight.com.