The head of TMX Group Ltd. said the faulty hardware that caused outages at its four stock and derivatives markets Friday was a freak accident – a glitch the maker of the equipment had never seen before.
Technical issues in a component that handles messaging to and from these venues forced TMX to halt trading more than two hours early. The Toronto Stock Exchange (TSX), TSX Venture Exchange (TSXV), TSX Alpha Exchange and the Montreal Exchange (MX) for derivatives were all affected, with clients losing access to order-entry functions. TMX replaced and tested the broken part, and all four exchanges opened Monday as usual.
Still, the episode is raising questions on Bay Street about the technology that underpins Canada’s biggest public markets and how much the TSX is relied upon for certain tasks, even as its market share has declined over the past decade.
Related: TSX says it’ll open as usual Monday after sudden trading halt Friday
On Monday, Lou Eccleston, chief executive officer at TMX, tried to reassure participants. He called the incident “an unprecedented event” because the backup system also failed. He declined to name the equipment vendor, but said the part has been installed at major companies all over the world and has been running inside TMX’s system for more than a year.
“The manufacturer tells us that they’ve never seen this in any installations anywhere in the world,” Mr. Eccleston said.
He added that having messaging for all four markets connected to one piece of equipment is “better for clients” because it means fewer connections, less complexity and lower costs. When asked why the backup didn’t kick in, he said: “You’d get into a technical discussion at that point. And I’m not a technology expert.”
Traders and technology experts on Bay Street questioned the design that led one piece of equipment to take down four markets and their backup facilities.
“Machines break – that’s something we all know,” said Sean Debotte, the CEO of Omega Securities. “The tough thing here is that these systems, the more you try to back them up and make them redundant, the more complicated they become, and those complications become prone to failing altogether.”
Mr. Eccleson defended the design: “If they saw the diagrams and the way it’s set up, it’s state of the art.”
He was quick to say what the incident was not: a cyberattack. There was no damage to the trading engines. And he said it had nothing to do with cost-cutting that took place earlier in his tenure.
“This is provided by a manufacturer who is state of the art. This is not from any cost-cutting measures,” he said. “We didn’t go with inexpensive or less-proven technologies.”
The MX is the only derivatives exchange in Canada, so options and futures trading ended early with nowhere else to print trades. Because there are other places where stocks can be bought or sold, equities kept changing hands even after the TSX shutdown – but volumes tumbled.
Before 2 p.m. Friday, 455 million shares of TSX- and TSXV-listed equities were traded in Canada on displayed markets, according to data compiled by the Neo Exchange. Just under 65 per cent of the action occurred on venues operated by TMX, while Nasdaq Canada handled 18 per cent and the Neo 16 per cent. The Canadian Securities Exchange (CSE) and Omega executed the other 5 per cent.
Once the three TSX markets went offline, the technology that brokers use to find the best way to execute a trade – called the smart order router – removed the affected exchanges from their ranking tables and started pushing orders to other Canadian venues.
Faulty equipment blamed
A faulty piece of equipment forced TMX Group Ltd. to halt its three equities markets more than two hours early last Friday. Here’s how Friday’s volumes compared with those of the previous day and where trades in these shares were handled before and after the outage.
Trading volume of TSX- and TSXV-listed stocks
Consolidated volume traded on displayed markets
by hour, in millions of shares
April 26
April 27
140
120
100
80
60
40
20
0
9 a.m. -
10 a.m.
10 a.m. -
11 a.m.
11 a.m -
12 p.m.
12 p.m. -
1 p.m.
1 p.m. -
2 p.m.
2 p.m. -
3 p.m.
3 p.m. -
4 p.m.
Where TSX- and TSXV-listed stocks traded on Apr. 27
Omega
NEO
Exchange
CSE
Nasdaq
Canada
3%
2%
14%
17%
Before
2 p.m.
TMX
64%
Omega
NEO
Exchange
8%
CSE
27%
16%
After
2 p.m.
Nasdaq Canada
49%
TMX
0%
the globe and mail, source: neo exchange
Faulty equipment blamed for stoppage
A faulty piece of equipment forced TMX Group Ltd. to halt its three equities markets more than two hours early last Friday. Here’s how Friday’s volumes compared with those of the previous day and where trades in these shares were handled before and after the outage.
Trading volume of TSX- and TSXV-listed stocks
Consolidated volume traded on displayed markets by hour,
in millions of shares
April 26
April 27
140
120
100
80
60
40
20
0
9 a.m. -
10 a.m.
10 a.m. -
11 a.m.
11 a.m -
12 p.m.
12 p.m. -
1 p.m.
1 p.m. -
2 p.m.
2 p.m. -
3 p.m.
3 p.m. -
4 p.m.
Where TSX- and TSXV-listed stocks traded on April 27
Omega
CSE
NEO
Exchange
NEO
Exchange
Nasdaq
Canada
Omega
3%
2%
8%
CSE
14%
27%
17%
16%
Before
2 p.m.
After
2 p.m.
Nasdaq Canada
TMX
TMX
49%
64%
0%
the globe and mail, source: neo exchange
Faulty equipment blamed for stoppage in trading
A faulty piece of equipment forced TMX Group Ltd. to halt its three equities markets more than two hours early last Friday. Here’s how Friday’s volumes compared with those of the previous day and where trades in these shares were handled before and after the outage.
Trading volume of TSX- and TSXV-listed stocks
Consolidated volume traded on displayed markets by hour, in millions of shares
April 26
April 27
Between 2 and 4 p.m., trading activity plunged
140
120
100
80
60
40
20
0
9 a.m. -
10 a.m.
10 a.m. -
11 a.m.
11 a.m -
12 p.m.
12 p.m. -
1 p.m.
1 p.m. -
2 p.m.
2 p.m. -
3 p.m.
3 p.m. -
4 p.m.
Where TSX- and TSXV-listed stocks traded on April 27
Omega
NEO
Exchange
Omega
CSE
NEO
Exchange
Nasdaq
Canada
3%
2%
8%
CSE
14%
27%
17%
16%
Before
2 p.m.
After
2 p.m.
TMX
0%
Nasdaq Canada
TMX
64%
49%
john sopinski/the globe and mail, source: neo exchange
Between 2 and 4 p.m., trading activity plunged, with just 62 million shares changing hands. Nasdaq handled almost half the volume, while the Neo did 27 per cent, the CSE 14 per cent and Omega 8 per cent.
TMX did not engage its disaster-recovery plans or reopen Friday because the technical issue took place late in the day and it wanted to ensure that its market data was accurate.
The outages could have been a lot more problematic had they occurred on a busier trading day either because of news, a rebalance of major indexes or the expiration of futures and options contracts.
But Mr. Debotte said the outages raise bigger questions about the way the Canadian stock market is structured and how much it still relies on the TSX for things such as closing prices. This figure is important for calculating the value of pension funds, ETFs and mutual funds.
“Should the last trade be the last trade, or should the only last trade that counts be the one at 1:39 on Friday afternoon at the TSX?” he said. “It makes absolutely no sense, considering all securities continued to trade.”
Another issue that was in plain sight on Friday is that many retail investors and their financial advisers were in the dark because they don’t pay for or have access to a data feed that consolidates a quote in real time and across all equities markets. As a result, these people wouldn’t have necessarily seen what was happening on other venues.
“When I see something like Friday and the impact it had due to the fact that so many people can’t see anything, it’s becomes obvious that this data has to be available,” said Jos Schmitt, the head of the Neo. “So let’s make it available and let’s tackle the cost issue.”