{"id":655767,"date":"2013-05-02T08:00:50","date_gmt":"2013-05-02T12:00:50","guid":{"rendered":"http:\/\/www.datacenterknowledge.com\/?p=96814"},"modified":"2013-05-02T08:00:50","modified_gmt":"2013-05-02T12:00:50","slug":"surviving-sandy-two-views-of-the-superstorm","status":"publish","type":"post","link":"https:\/\/mereja.media\/index\/655767","title":{"rendered":"Surviving Sandy: Two Views of the Superstorm"},"content":{"rendered":"
\"sandy-house\"<\/p>\n

A look at some of the damage wrought by Superstom Sandy on a property adjacent to the IFF data center in Union Beach, New Jersey. (Photo: IFF)<\/p>\n<\/div>\n

LAS VEGAS <\/strong>– For Alex Delgado, things were going from bad to worse as Superstorm Sandy slammed the Jersey Shore. It was high tide, during a\u00a0full moon. There was a 13 foot\u00a0storm surge, and the data center was less than a mile from the beach. Six hours into the storm, the company\u2019s operations team in India had to be evacuated due to a cyclone.<\/p>\n

The staff at the International Flavors & Fragrances (IFF) data center in Union Beach, N.J. used to joke about a single telephone pole that carried “half of the Internet and half of its power.” As Sandy came ashore, that was the pole that fell. In short, had Delgado won a raffle that week, it would have been for the Hunger Games. Everything was going wrong.<\/p>\n

The campus was swamped\u00a0with six to seven feet\u00a0of water. Both its power substations were under water, as were\u00a0the diesel fuel pumps. UPS batteries were nearing their end of life. Street power was out, and access to the facility was hindered by partially collapsed road.<\/p>\n

Different Scenarios, Different Considerations<\/h3>\n

Delgado, the Global Operations and Data Center Manager for IFF, shared his experience this week as part of a keynote panel at Data Center World in Las Vegas.\u00a0The panel showcased\u00a0two stories of Sandy’s impact:\u00a0one\u00a0from the Jersey Shore at the heart of the damage, another\u00a0from\u00a0Philadelphia.<\/p>\n

The\u00a0data center in Union Beach supports more than 50 manufacturing facilities around the world for IFF,\u00a0 a chemical manufacturing company that did over $2.8 billion in revenue last year. While Delgado and his team struggled with the storm, the\u00a0event had no major\u00a0impact on\u00a0customers, as the company didn’t lose a single order.<\/p>\n

The 4,500 square foot facility is a\u00a0single tenant building with 30 minutes of UPS backup.\u00a0Its disaster recovery site is 2 hours away at an IBM facility in\u00a0Sterling Forest, New York. As the storm intensified, IFF was able to shift its critical operations to the backup facility.<\/p>\n

The damage in Union Beach was severe. The data hall stayed dry, as it was on the second floor of the building. But the storm surge took out power and mechanical infrastructure, and flooded the machine shop,\u00a0ruining most of the facility’s power tools and spare parts. With the power out and roads closed or blocked, staff stayed in place for days.\u00a0With provisions exhausted after the first 48 hours, IFF staff subsisted on vending machine food\u00a0as they began the recovery effort, Delgado said.<\/p>\n

The data center was returned to service on Dec. 8 with new generators and infrastructure. Delgado wound up procuring\u00a0300 batteries and 3 generators.<\/p>\n

Delgado’s\u00a0key “lesson learned ” included vendor support.\u00a0”If\u00a0you don\u2019t have a good relationship with your vendors, start shaking some hands today,\u201d he said. He also noted that\u00a0the company had moved to cloud email, which saved a ton of headaches in terms of communication.<\/p>\n

The View From Philly<\/h3>\n

Donna Manley, IT Senior Director at the University of Pennsylvannia, showed a different side of the storm. While Philadelphia wasn\u2019t nearly as impacted as much, the operational impact of the storm was great.<\/p>\n

The university’s data center is in a multi tenant building, with a main data center of 4,850 square feet in\u00a0the University City section of\u00a0Philadelphia. Manley\u2019s story is important because it revealed a larger concern than just the data center: the\u00a0city of Philadelphia\u2019s aged infrastructure.<\/p>\n

A week prior to the storm, Manley and her team started the planning process.\u00a0They identified teams, began tarping the windows, and put diisatser recovery\u00a0provider SunGard on alert. \u201cWe started our crisis command center on the 29th, setting up a separate box.net instance just in case we lost power and were in an emergency situation,” said Manley.<\/p>\n

Understanding the geographic diversity of the staff was important, as some employees lived in areas where the storm hit hard. “We had very few individuals that could have been on site,” said Manley. “\u201cWe needed to make sure there was technical and management staffing.””<\/p>\n

Cloud Services Play a Role<\/h3>\n

Manley leveraged online storage provider Box.net<\/strong> to get them through the storm. “Resourcing doesn\u2019t just mean people,” said Manley. “One of the big things we have going on is our documentation. Up until recently, we had it in Sharepoint. We made it available on box.net, and we didn\u2019t have to worry about servers going down and documentation not being available to us.”<\/p>\n

Manley\u2019s advice is to have a data center crash kit checklist. “Because we\u2019re an urban campus, we have a couple of unique items on there – respirator masks, subway tokens to get to disaster recovery site at Sungard)”, she said.<\/p>\n

She said it’s\u00a0also important to\u00a0read the fine print on Disaster Recovery Agreements to see whether a fee is required to put your provider on standby. There’s also\u00a0food, as\u00a0there’s\u00a0 a chance workers will have to stay put at the data center for extended periods of time, and the local restaurants aren\u2019t as committed to staying online as a data center.<\/p>\n

Both\u00a0organizations\u00a0said the prospect of managed services and hosting now appealed to them a little bit more than prior to the storm. Cloud services played an important role in both disaster plans, even if only to keep communications open through email.<\/p>\n","protected":false},"excerpt":{"rendered":"

A look at some of the damage wrought by Superstom Sandy on a property adjacent to the IFF data center in Union Beach, New Jersey. (Photo: IFF) LAS VEGAS – For Alex Delgado, things were going from bad to worse as Superstorm Sandy slammed the Jersey Shore. It was high tide, during a\u00a0full moon. There […]<\/p>\n","protected":false},"author":7436,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-655767","post","type-post","status-publish","format-standard","hentry","category-news"],"_links":{"self":[{"href":"https:\/\/mereja.media\/index\/wp-json\/wp\/v2\/posts\/655767","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mereja.media\/index\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mereja.media\/index\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mereja.media\/index\/wp-json\/wp\/v2\/users\/7436"}],"replies":[{"embeddable":true,"href":"https:\/\/mereja.media\/index\/wp-json\/wp\/v2\/comments?post=655767"}],"version-history":[{"count":0,"href":"https:\/\/mereja.media\/index\/wp-json\/wp\/v2\/posts\/655767\/revisions"}],"wp:attachment":[{"href":"https:\/\/mereja.media\/index\/wp-json\/wp\/v2\/media?parent=655767"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mereja.media\/index\/wp-json\/wp\/v2\/categories?post=655767"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mereja.media\/index\/wp-json\/wp\/v2\/tags?post=655767"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}