Planning enables central stations to survive Sandy
Nothing sets off alarms like a natural disaster and Hurricane Sandy, which slammed the East Coast late last month was an example of the power of central station monitoring response in action. But for the most part, monitoring centers responded well to the call to action.
“We got killed by traffic,” said James McMullen, president and COO of COPS Monitoring, one of the nation’s largest monitoring service providers. COPS is headquartered right at the heart of the storm in Williamstown, N.J., but has operations in Arizona, Florida, Tennessee and Texas, in addition to its New Jersey headquarters. Those monitoring centers proved lifesavers.
During and after Hurricane Sandy, the work floor at most Northeastern central stations looked like Christmas morning with lights flashing, bells ringing and workers scurrying about to answer all the requests coming in. The storm, which battered the East Coast, made landfall near Atlantic City. Sandy, exacerbated by high tides and high winds and a cold front coming from the west, dumped ungodly amounts of rain and water on all the cities in the Northeast. Hardest hit were areas like Philadelphia, New York, Hartford and Boston. But Hurricane Sandy’s impact was felt as far inland as Cleveland, Ohio where power outages were commonplace, trees were toppled and alarms rang constantly at monitoring centers and even in Chicago, where Lake Michigan recorded 20-foot-plus waves.
“If anyone says they didn’t have problems ask them if they know the meaning of BS,” McMullen challenged. Response times skyrocketed from 20 seconds to over six minutes in some locations. However, everyone knew the situation was critical. McMullen gratefully recalled several dealers who told them simply to log reports and not respond. Towns like Beach Haven, N.J. were underwater. Even police were not responding.
President Obama declared swaths of New Jersey, New York and the mid-Atlantic disaster areas. Reed Exposition postponed the ISC East trade event planned for the week due to the State of Emergency declared by New York’s governor and the potential danger presented by Sandy. Yet most central stations, while chaotic, remained operational.
“Things went well but that’s a relative term,” said Pamela J. Petrow, CEO of Vector Security, based in Pittsburgh. Like other firms with multiple locations, traffic was rolled out of New Jersey and to offices in western Pennsylvania and Ohio. Workers had plenty of notice and the office building at Cape May Courthouse, N.J., was boarded up in advance of the storm. Vehicles were moved out of low, flood-prone parking lots to higher ground.
“Our centrals all are redundant. We moved all of our calls out of the affected area,” Petrow said.
Downed trees and flooded areas prevented trucks from getting everywhere they wanted to be. In some areas, police stopped taking calls for any issue other than fire, EMS or similar life safety.
“All key holders were notified,” Petrow added.
While COPS was in the heart of the storm, having multiple locations allowed them, too, to move alarm receiver traffic to Arizona and Florida. Their mainframe computer in New Jersey was backed up in Las Vegas.
Every central station has battery back-up and on-site power generators to assure they remain up and running in the face of a disaster. Complete Security Systems, based in Marlboro, N.J., closed its office during the storm and they remained closed for several days after the storm came through.
“It was busy but we are the storm troopers,” said Morgan Hertel, vice president of operations for Syracuse, N.Y.-based Rapid Response Monitoring. Rapid Response handled almost one million alarms in the critical 24 hours of the storm.
“We were prepared,” Hertel said. At the peak of the storm Rapid Response had between 150 and 160 workers. They are fortunate in having headquarters in Syracuse. Although they serve a nationwide customer base, the Syracuse station was not seriously affected by the storm and there was no need to roll operations to a different location.
“We had two client central station companies that were on full disaster recovery stand by,” said Melissa M. Courville, head of marketing at Dice Corp. in Michigan. One company was 80,000 accounts in size and the other 50,000.
“These companies were processing overflow signals through the DICE Disaster Recovery Center,” Courville explained. “This methodology happens when clients don't need to do a full fail-over but their signal transmissions to the DICE DR allow them to send any overflow to us to grant them additional bandwidth to handle the extra signals they're taking on for processing.” Bay City, Mich., where DICE is headquartered, was in a safe harbor. While the East Coast was reeling, Bay City had a light drizzle and cloudy climate, not much different than any other late October day.
The day following the brunt of the storm, Dice had another central fail-over. This involved about 100,000 accounts. Courville said that incident was due to telephone line outages. The company immediately started the process of rolling over the IP infrastructure aspect of their business, which is another 40,000 accounts, she said.
That brought the figures to approximately 270,000 total accounts running on the DICE Disaster Recovery Center. “Our DICE DR team worked around the clock to ensure that our client profiles were already pre-loaded and ready to go in the impact zone,” Courville said.
Other client companies that are hot-redundant on DICE, with full redundancy capabilities, failed over to DICE and back again without DICE intervention as they are automatic.
By late Tuesday, long-distance carriers like AT&T and Verizon were having trouble getting toll-free traffic out of New England. The COPS staff discovered it could move traffic from New England to the West Coast and rerouted everything that way. Knowing what was possible and having a plan to implement it was a life-saver everywhere.
Going proactive
Mason Monitoring, located in Kings Park, N.Y. went pro-active right away. Mason sent an email blast to its customers acknowledging that Hurricane Sandy had ravaged the area. “The early reports indicate that this Super Storm has caused wide spread damage to many major cities leaving millions without power,” Mason’s website said even before the rains had stopped. “Through it all we’re proud to say that Mason is here…undeterred and on the job.
“While our neighborhood sits darkened and powerless in Sandy’s wake our central station remains a literal beacon of light,” Mike Cannatella told customers. ”Our back up plan in full effect, we march on…it is what we do. It is that kind of reassuring message, coupled with a subtle sales pitch reinforcing quality and resiliency that keeps customers calm.
“We were up 100 percent. We didn’t miss a heartbeat,” Cannatella added. Mason serves about 40,000 customers and much of their customer base in Connecticut was without power. This resulted in many air-conditioning failures and other alarms. Mason brought in a few more workers to handle the alarms.
Ironically, Mason had just posted a notice to its website a few weeks earlier. In August, their service area suffered a major power outage due to Hurricane Irene. Thanks to a brand new 100 kilowatt (412 continuous amps) John Deere diesel generator with a 400-gallon tank, they were unaffected. For extra redundancy and added peace of mind, they signed a four-hour service contract to replace or service that new generator. “Should that circumstance ever arise, our 10-hour battery backup supply will hold us for almost three times that duration,” the website trumpeted. Little did they dream that just two months later, they would be testing the system to its limits.
Mason also has a smaller, gas-powered unit. Cannatella figures they could be down for a month and still maintain service. Backup and UPS is part of being in the central station business, Cannatella said, noting that it is just like having a spare tire in the car.
At the most basic level, a 6 kilowatt generator could keep 20 computers running. However, most central stations go much bigger.
Post-mortem diagnosis
For all of the dealers in affected areas, it is too early to have conducted post-mortem debriefings. “There will be an impact,” Petrow said, noting that no new installs have been done for some time. Sales people were handling other chores, so that end of the business is going to take a hit. Operationally, however, Vector was handling all calls even though one central station remained on generator power for almost a week.
“We had our storm policies and procedures in place. They worked in the past. They worked this time,” Hertel said. “The only thing we’d do different is order in a bigger variety of pizzas.”
In the days and weeks after the storm, Rapid Response was dealing with the aftermath of the storm. Despite the massive number of alarms, Hertel said they did not have to clear anything and nothing got blown.
“We are used to dealing with local thunderstorms,” McMullen said. “This storm covered 1,000 miles.”
Another key is to have staff prepared to get to the station to handle the alarms coming in.
Given the scope of things, central stations faced a double-whammy. Alarm traffic was outrageous. Some police departments simply stopped answering phones or put security firms on hold. “They were going crazy,” McMullen acknowledged.
On top of that, calls came in from people asking whether there was an alarm at their vacant vacation property at the shore. Those customers who got through to the central station often wanted to chat about things they were experiencing: blackouts, transformers popping, trees falling.
McMullen said one thing they will do differently in the future is to post a greeting asking people with normal service or billing questions to call back later or to access the data on the web on their MPower system or through their MPower-ME, the mobile edition that allows retrieving data by cell phone.
“We will make little tweaks to the pre-message,” McMullen said. He had nothing but praise for the dealers and suppliers who pulled together during the crisis.
“Our thoughts go out to the families and businesses affected during this time, as well as our condolences for the loss of life that came with the surge of the storm, and as those brace for the devastation of the aftermath—we continue to stand ready to support and serve,” Courville said.
Petrow said that employees’ lives were affected. “Our people were wonderful,” she said.
While most service providers echoed Courville’s sentiments, it is thorough planning and actions ahead of an event that make a difference. Planning is the key to handling any such outage. One area that is important is having relationships with technical vendors who are on standby during critical times.
