Rogers says software glitch led to massive wireless outage

Share

Rogers Communications Inc. said a software glitch created a big spike in "signalling traffic" that caused one of the worst wireless network outages in the company's history.

Canada's largest wireless carrier determined that root cause on Thursday roughly 18 hours after implementing a fix that restored voice and text services for customers across the country.

The vast majority of the company's 9.42 million wireless subscribers lost those services Wednesday, starting around 6:00 p.m. ET. The software fix was employed around 10:30 p.m., which enabled a full restoration of services a short time later. Data services were not affected by the outage.

Although the problem lasted for less than five hours, the furor that erupted over the service interruption, particularly on social media sites like Twitter, reflects how dependent Canadians have become on their smartphones.

A software glitch on a switch caused the surge in signalling traffic on one of two networks the company runs in parallel to provide cellphone service. While one network carries voice traffic, a second network is used for "signalling" or the instructions for handling calls. For instance, signalling is used to ensure that a call does not drop if a customer is moving while talking.

"There was a big spike in signalling traffic and what happened is the mobile switches didn't handle it properly, and they basically restarted. And when they restarted, it caused the service interruptions," said Bob Berner, executive vice-president of network and chief technology officer.

Some switches took longer than others to reset. The software fix should prevent any further service interruptions because it changes how switches handle a signalling surge, the company said. There is no indication that the outage was caused by hacking or a security breach.

"We've been doing this for 28 years and we've never seen anything like this happen before. I don't expect it to happen again," added Mr. Berner.

Telecom consultant Mark Goldberg questioned whether the federal telecom regulator should be collecting and sharing information on network outages to assist consumers.

Canadians are among the world's heaviest users of smartphones and text messaging is extremely popular. Canadians send more than 270 million text messages per day, with a total of 96.5 billion texts sent in 2012, according to the Canadian Wireless Telecommunications Association.

"It has become part of people's lives," added Mr. Berner. "If you just watch people anywhere on the street, everyone is always doing something on their wireless device."

Moreover, an increasing number of Canadians are ditching their home phones altogether with roughly 12.8 per cent of Canadian households considered "wireless only" in 2011, according to the latest data from the CRTC. There are also fewer payphones available for Canadians to use when cellphone networks fail.

Rogers CEO Nadir Mohamed, meanwhile, apologized to customers for the inconvenience.

"I recognize this service interruption was unacceptable for our customers," Mr. Mohamed said. "We worked as quickly as possible to restore service and it was gradually restored over the course of the evening."

Rogers and its Fido subsidiary said they would credit postpaid wireless subscribers for one day of service because of the outage. It is unclear on how much that would cost the company in lost revenue.

Since the Rogers' corporate website received a surge in traffic on Wednesday evening, some customers complained they were unable to pay bills or check their Internet usage. The company is advising customers to call if they have concerns about their bills.

Rogers also advised customers to use land lines if they needed to reach police and other emergency services on Wednesday evening.

Many customers took to social media to complain about losing service, and some of those complaints were misdirected. Glenn Rogers, who lives in Brooklyn, N.Y., and uses the @rogers handle, saw his Twitter feed inundated by angry customers.

"The wrath of a thousand Canadians is a mighty sight," he tweeted. He added: "Thanks for all the kind tweets Canada. May they never take your phone or internet from you again."