There are a few occasions during which information systems have to function absolutely perfectly, with no margin for error and zero tolerance for downtime: military invasions, open-heart surgery, major financial transactions — and Super Bowl Sunday.
In preparation for this weekend’s game, the National Football League has moved more than 250 staff members from the league’s Manhattan headquarters to a field station in New Orleans. The NFL rented 60 rooms in the Hyatt Regency and removed the beds and bedroom furniture, replacing them with desks, office chairs, fax machines, and phones, plus 200 Dell PCs, three server clusters, a local area network, and several T-1 lines connecting everything to the New York office.
The NFL’s IT staff of 15, led by senior IT director Dave Port, built the network by stringing network cables from room to room around the outside of the building, via the balconies. Because the rooms’ existing electrical systems couldn’t handle the load, the Hyatt supplied additional power through a handful of 200-amp power panels set up on plywood boards in the hallways. When Port’s staff finished their work, the rooms looked like offices, with each containing four or five desks, several PCs, a laser printer, a fax machine, and business phones. The whole project was accomplished in about three weeks. To make it all happen, Port’s staff worked from 7 a.m. to midnight, seven days a week. “We don’t get to see much of the sights down here,” Port deadpans.
All of this computing firepower is in place to make sure that the NFL can supply game statistics, play-by-plays, and background information to a variety of recipients: the league’s websites at NFL.com and Superbowl.com, the computer systems back in the NFL’s home office, the members of the media covering the game, and the league’s broadcast partners, Fox (FOX) and CBS (VIA).
“We’re in the entertainment business,” says Port. “We treat this as no different than a network treats its broadcast. This is the biggest sporting event in the world, and we want to make sure all of our resources are available to us and to our partners.”
Port has been building onsite IT systems for the NFL for the past 17 Super Bowls; each is bigger and more complicated than the one before, he says. To safeguard against disaster, every important system in New Orleans has at least one spare — a safety measure known as redundancy, in techno-speak.
For example, the NFL’s servers are Dell PowerEdge 6450 systems, featuring multiple CPUs, hard drives, and power supplies, so that if one of those components goes on the fritz, the server can keep running. In case that’s not enough, each server is paired with an identical machine that’s on hand to take over if the first one goes down. This technique, called server clustering, helps to guarantee that critical business applications — like the league’s e-mail system — keep running, no matter what.
“A lot of our communication depends on e-mail, because the phone systems get bogged down,” says Port.
Likewise, Port doesn’t rely on just one T-1 line — he’s got several, to make certain that the New Orleans staff can stay in touch with the rest of the world even if one of those connections gets cut by, say, a wayward backhoe or a particularly enthusiastic tailgate party.
After Sunday all the equipment will be pulled out in just two or three days. The servers will head back to New York, where they’ll assume more prosaic roles in the company’s business systems. The hundreds of desktops will be sold to NFL teams for use in their offices. And someone will have to put all the beds back into the Hyatt’s rooms.
What’s the lesson here for other businesses? Redundancy is essential for any component of your IT infrastructure, particularly when you’re managing high-value, high-profile, time-critical events. Careful planning is essential to success. And despite what your own IT staff may say, building (or rebuilding) an entire office for hundreds of people can be accomplished in just a few weeks — provided that you’ve got the budget, the staff, and the skills to do it.
Link: Network Defense for Super Bowl Sunday
Link broken? Try the Wayback Machine.