The company responsible for a mass web outage that hit three of Australia’s big four banks, Virgin and Australia Post, among others, has said a routing table error was to blame for the service disruption, not a cyber attack.
A wide range of online services were interrupted on Thursday afternoon following a widespread outage that affected ANZ, Westpac and the Commonwealth Bank, as well as other businesses such as Virgin Australia and Australia Post.
Customers took to social media to report troubles accessing bank services online or through banking apps, with some stuck unable to pay for goods or services.
NAB was the only one of the big four banks not to experience any disruption, along with many of the smaller banks and credit unions.
However, the problems went far deeper, with staff at some of the major banks unable to access the internet and the Reserve Bank’s website and other systems also down for a period of time.
“We have implemented appropriate mitigations and the website is now back up and running,” the RBA said.
However, the ABC understands some market operations between the Reserve Bank and commercial banks were cancelled because of the technical problems many of the participants were facing.
Although it is understood that end of day financial settlements between the major banks were carried out as usual.
The disruptions appeared to extend beyond Australia, with the Hong Kong Stock Exchange’s website briefly down along with reports of dozens of other website outages, including for some US airlines.
Akamai blamed for outage
The source of the problem was an outage with service provider Akamai.
Akamai, is a US-based global content delivery network or “edge platform”, cybersecurity and cloud service provider.
The company promotes itself as “the world’s largest and most sophisticated edge platform” to protect businesses and users from online threats while delivering content quickly.
After repeated attempts to contact the company’s Australian and Asia-Pacific representatives on Thursday, ABC News finally made contact with one of its US spokespeople who directed us to a blog post the company released overnight.
The post confirmed that problems began at 4.20am UTC (2.20pm AEST) on Thursday and affected approximately 500 customers using version 3.0 of Akamai’s routed Prolexic distributed denial of service (DDoS) protection system.
“The issue was not caused by a system update or a cyber attack,” the company said.
“A routing table value used by this particular service was inadvertently exceeded. The effect was an unanticipated disruption of service.”
Akamai said many of its customers had services restored quickly.
“Many of the approximately 500 customers using this service were automatically rerouted, which restored operations within a few minutes,” Akamai said.
“The large majority of the remaining customers manually rerouted shortly thereafter.”
However, it took between half an hour and a couple of hours for the big banks’ services to be fully restored.
Akamai said the affected service was restored by 8.47am UTC (6.47pm AEST), and customers began the process of routing back on to the service at that time.
An outage at a different CDN, Fastly, was behind the recent shutdown of major global news and media sites, including the BBC, Financial Times and Reddit.
‘Every man and his dog uses Akamai’
Edith Cowan University’s associate dean for computing and security, Paul Haskell-Dowland, said Akamai was one of the biggest and oldest CDN providers.
“Akamai is a very well known CDN provider that nobody knows about,” he quipped.
“Akamai has been around a bit longer than Fastly.”
He said its services were ubiquitous, meaning that many other less well-known businesses were likely to have been affected by the outage.
“Every man and his dog uses Akamai,” he told ABC News.
Paul Haskell-Dowland said the most likely cause of the outage would be a software or configuration problem at Akamai, which does not happen very often.
“We really don’t hear of that many major outages, so they’ve been pretty reliable,” he said of CDNs.
However, he also noted that the recent well-publicised Fastly outage along with Thursday’s problems may draw the attention of hackers and other malicious actors.
“Perhaps it’s a bit more on the radar of some of our adversaries,” he said, noting that it can be extremely difficult for a user to switch quickly to another delivery platform if there is an outage at its CDN provider.
Companies took to social media to inform their customers of the problem, even as they themselves were scrambling to figure out what had gone wrong.
“We’re aware some of you are experiencing difficulties accessing our services and we’re urgently investigating,” CBA wrote on Twitter.
Virgin Australia tweeted that it was experiencing a system outage that was impacting its website and contact centre.
Australia Post was also affected, describing the issue as an “external outage” affecting a number of its services.
By 4pm AEST, services were starting to be restored, with some bank customers able to log in to their apps.
Shortly after 5pm, Virgin confirmed it was back up and running.
“Virgin Australia is pleased to confirm that an IT outage has been resolved and our website and guest contact centres are operating as usual,” the airline said in a statement.
“Flights are largely operating as scheduled and we apologise to any guest who has been inconvenienced.”