What a day! Yesterday morning, the New York Stock Exchange went offline, United Airlines grounded its entire fleet, and the Wall Street Journal website went offline.
All three outages were related to technical glitches of one sort or another. And on top of that, the Chinese stock market tanked. That wasn’t a technical fault, but it didn’t stop people from lumping it together with the other three big news events of the day, adding in some horrendous commutes in New York and San Francisco and saying the sky is falling.
Or, maybe, don’t panic: These are all unrelated events, and coincidences happen.
Then again, maybe we should panic: These are symptoms of how badly software sucks, and it’s only going to get worse.
It’s tempting to join the chorus of extremist headlines, but I think it’s more productive to try to draw some useful lessons from the day.
First of all, the three technical outages were not related to hacks or cyberterrorism or any other malicious activity. The head of Homeland Security spoke with both United and the NYSE, and attested that the glitches were “not the result of any nefarious actor.” United blamed its hourlong outage on a “network connectivity error.” The WSJ similarly denied being hacked.
That said, cyberattacks do happen. Already in 2015 we’ve seen a few gigantic, disturbing hacks: The State Department was breached and lost control of millions of personnel records, including Social Security numbers; and Anthem Blue Cross got hacked and lost control of tens of millions of customer records. That’s following a horrendous 2014, in which Sony Pictures and JP Morgan got embarrassingly hacked, among many others. If you really want to scare yourself, check out Paul Sparrow’s Hackmageddon site, which collects timelines of all reported cyberattacks, month by month.
Lesson 1: You need to invest more in security. If the prospect of a technical glitch taking you offline for an hour or a day scares your CEO, just imagine how she’ll flip out if hackers abscond with millions of accounts, and the story gets written up in the WSJ — or VentureBeat.
Second, yesterday’s technical problems had no lasting effects. Although the NYSE is one of the world’s biggest stock exchanges, it only handles about 20 percent of the U.S. stock market. While it was down, other exchanges, like the NASDAQ, remained online. As a result, the nearly four-hour outage had almost no effect on the stock market.
United’s system crash caused a lot of grumbling among its customers, but the outage didn’t last very long and most of those customers probably reached their final destinations with only slight delays. The brand damage would have been much less if United didn’t already have a reputation among many people for indifferent customer service.
As for the WSJ, its outage came at a bad time. I can tell you from personal experience that journalists start tearing their hair out if their own site has technical problems at the same time they’re trying to cover a big news story. I can only imagine the yelling that was happening in the Journal‘s newsroom as the team discovered their site was borked at exactly the moment they were trying to cover United and NYSE. Fortunately the company quickly put up an alternate homepage, and got things rolling again in less than an hour. Today, things are back to normal on the site, and I’m sure the WSJ’s traffic suffered only a temporary dip yesterday.
Lesson 2: Big outages are survivable — if you’re prepared. Put in a disaster recovery plan and keep it up to date.
Third, don’t believe everything you read in the press. (Yes, I realize the irony of a journalist saying this.) But know that you may have to deal with the press and the public, even if their perceptions are wildly off base. Consider yesterday’s outages as a case study of what happens during an outage. Headline writers get excited, Twitter blows up, and suddenly you’re dealing with angry hashtags and aggressive inquiries from financial and business reporters.
Lesson 3: Make sure your disaster planning includes communicating with your customers (and the press) as quickly and openly as possible. Be on Twitter. Be on Facebook. Respond as quickly and as humanely as you can to angry customers and confused journalists, and you may be able to contain the damage.
The bottom line: Don’t panic. Don’t read too much into three big outages happening in a single day.
But do take these outages as a serious warning to take care of technical debt, invest in security, and make sure you have backup servers ready in case something really goes sideways.