Research In Motion Ltd.'s co-chief executive officers personally apologized for the worst system failure in the history of BlackBerry smart phones, as RIM engineers finished correcting the problem that caused tens of millions of customers to suffer service outages for three days.
"Since launching BlackBerry in 1999, it's been my goal to provide reliable real-time communications around the world," Mike Lazaridis said in a video that was posted to RIM's website early Thursday morning. "We did not deliver on that goal this week, not even close. ...You expect better from us, and I expect better from us."
The apology, and the rare appearance of Mr. Lazaridis and co-CEO Jim Balsillie together at a press conference, marked a change in RIM's approach to the situation. The company has been criticized by customers for not giving customers enough information about the problem and the repair effort. As late as Wednesday evening, after the service interruption had spread to North America, neither CEO had spoken publicly about it, and RIM went several hours between giving updates on the status of its network.
But by late Thursday morning, RIM said it had fixed the server issues that caused the outages – which began in Europe but soon spread to BlackBerry users around the globe. And with service returning to normal for the vast majority of RIM's 70 million customers, talk soon turned to the issue of who will pay to compensate them for frustrating e-mail downtime.
Rogers Communications spokeswoman Carly Suppa said the company is currently "exploring options with RIM" on reimbursing clients, but could provide no more details. Mark Langton, a spokesman for Bell Canada, said the compensation issue was "the question of the day" because some European and Middle Eastern carriers have already decided to reimburse customers. Mr. Langton said he could not comment on whether Bell will do the same.
Other carriers – notably Vodafone Group PLC, the world's largest cellphone carrier – have raised the possibility of asking RIM for compensation. But when asked about the issue during a conference call with reporters, Mr. Balsillie said that now is not the time for "finger pointing," and added that many carriers worked with RIM to help solve the outage problems.
While the technical glitch isn't likely to cause any significant financial damage to RIM, it came at a poor time for the company, which has lost a great deal of market share in the past 12 months, particularly in the United States.
More details emerged Thursday on what happened. The first problems arose on Monday morning, and both the company's CEOs were immediately notified. In the three days that followed, Mr. Balsillie, scheduled to attend a conference in Dubai, travelled around Europe and the Middle East to talk to wireless company executives, and Mr. Lazaridis stayed in Waterloo, Ont., to head up the engineering teams trying to solve the glitch.
The problem at the European server originated in a device called a core switch, which acts as a sort of traffic director for BlackBerry data, sending each message to the right location. Core switch failures have happened before, but normally a backup system kicks into gear to keep the server running smoothly. On Monday, the backup switch also failed. As a result, other parts of the BlackBerry network began to overload.
Over the next two days, a backlog of messages began building up around the globe. The backlog got so bad that RIM began deliberately slowing down data traffic in Europe in order to ease the pressure. Even after RIM engineers fixed the core switch failure, a backlog remained, and some users were still seeing e-mail delays as of Thursday.
It wasn't until overnight Wednesday, after he was confident the technical problems had been fixed, that Mr. Lazaridis recorded a short video at the company's Waterloo headquarters, apologizing to customers for the outages. The video was posted a short while later, early Thursday morning.
The question in some observers' minds, however, is whether the co-CEOs' mea culpa came too late to appease angry consumers.
"I think the video that came out [Thursday] they should have had that on day one," said Queen's University marketing professor John Pliniussen.
"It's sad that a world-class company with state-of-the-art technology has state-of-the-ark public relations."
With files from reporter Richard Blackwell