Statscan chief fires back at household survey critics

Share

Canada's first National Household Survey has been a "success," and those who criticize the quality of its data are doing a "disservice to Canadians," Statistics Canada's chief statistician says.

His comments come ahead of the next batch of survey results, to be published Wednesday, which will focus on labour and education trends in Canada. The first wave of data, on aboriginal and immigrant populations, generated debate over data quality and the usefulness of the results, particularly around data at the local level.

Statscan itself acknowledges that information for 1,100 census subdivisions – including almost half of municipalities in Saskatchewan – was withheld in the first instalment of data because of low response rates, while the population counts for some groups, particularly Filipinos, appear to be overestimated.

But it's misleading to say the data is unreliable, said Wayne Smith, chief Statscan statistician in an interview with The Globe and Mail. "It's irresponsible to try and dissuade Canadians from using what is an extraordinary rich and powerful database. To make them nervous about that is I think irresponsible."

He acknowledges the data from the NHS is not of the same calibre as a mandatory census, but says that doesn't rule out its usefulness. "True, it's not a census, true, there's been some loss of small-area data and true there's more volatility in the estimates for small populations and small areas, particularly small populations in small areas. But the data turns out to be remarkably strong," given the agency's ability to use external information and benchmarks to help verify the results.

The federal government cancelled the long-form mandatory census in 2010, saying it was too intrusive, and replaced it with a voluntary survey. The abrupt move was controversial partly due to concern the data wouldn't be as accurate because some groups, such as recent immigrants and low-income people, are less likely to respond.

The survey wound up costing $22-million more than what the former census would have cost, largely stemming from extra printing and field costs associated with "chasing non-respondents," Mr. Smith said.

It's time for researchers to "get over" comparing the results to what a mandatory census would have yielded, Mr. Smith suggested. "That ship's sailed," he said. "The issue now is, are we going to do a disservice to Canadians by trying to persuade them that this data is somehow flawed, when clearly it is not, and clearly it is more robust than anything else out there, from anybody, that is coming from a voluntary survey.To label that data as unreliable is to mislead people in ways that are simply unfathomable and are certainly not scientifically supportable."

Robert Gerst, partner in charge of statistical methods at Converge Consulting Group Inc. in Calgary., has said the NHS data is "worthless" in part because it's no longer a random sample. Manitoba's chief statistician Wilf Falk is concerned about how representative the sample is and says it's tough to know which data is good and which isn't.

The data is probably solid at the provincial and national level, "but start driving it down [to the local level] and who the hell knows?" said Mr. Falk, who has been Manitoba's top statistician for 34 years. "For smaller communities, it's a problem."

He uses census data to prepare profiles on each of the province's 57 ridings on who constituents are and demographic changes in each – aimed to better inform members of the legislature on who they're representing and what their priorities are. Now he's not sure if he can still do that. In Manitoba, data wasn't released for about a third of communities because they didn't have high enough response rates. Now, "you've got holes in your data."

Attention is now focused on the format for the 2016 census. A decision by the federal government is expected to be made later this year or early next year, Mr. Smith said.

Beyond 2016, Statistics Canada and other agencies are looking at more affordable ways to produce information, such as using administrative data.

Mr. Smith said he understands the tradeoff between data quality and privacy concerns, and is willing to conduct the National Household Survey again if that's the federal government's decision. "I believe that we can substantially meet the requirements of Canadians through a repeat of the version of the 2011 census."

This week's NHS release will contain information on the most popular jobs in Canada, where we work and how we get there. It will also explore languages spoken in the workplace and trends in educational attainment.