The Power of PSIM Protects Utility Giant’s Critical Infrastructure
Headquartered in Charlotte, N.C., Duke Energy is one of the largest electric power companies in the United States, covering a service territory of approximately 104,000 square miles in the Carolinas, Midwest and Florida, and supplying energy to about 7.2 million U.S. electric retail customers. The Enterprise Security Command Center (ESCC) is the focal point for Duke’s security operations. Here, Duke Energy facilities spread across the company’s vast service territory are centrally monitored to ensure the safety and security of Duke’s critical infrastructure, assets and employees.
CriticalSpace Solutions, a Georgia-based company that specializes in collaborative visual communications, provided the original programming and conceptual designs, and integrated all of the audiovisual solutions in the ultramodern command center, which was commissioned just last year. The centerpiece of the command center is a gigantic 40-foot video wall that can display real-time or recorded video, weather maps, websites, operator workstations mapped to it — virtually any visual information needed for situational awareness. It has eight sections, each of which can be subdivided into four to 16 customizable viewing panels.
“The things that were strikingly evident to me when I went in there the first time were the ergonomic design, the functionality of the workstations, and their relationship to the video wall,” says Darren Myers, managing director of Enterprise Protective Services for Duke Energy. “Everything is very well situated so that the operators can sit in a comfortable environment for long periods of time, and supervisors can have a clear view of operators and the activity displayed on the video wall.”
Another innovative feature of the facility is its collocated workspace for technical support staff, partitioned from the main command center by a wall. “If we encounter any problem with our equipment, the support staff is right there; they can quickly determine what’s going on and resolve it,” explains Rocco A. Marcello Jr., who as former director of Duke Energy’s Enterprise Protective Services, spearheaded the project. “There’s no lag in communication.”
Adjacent to the command center floor is a large conference room, also designed by CriticalSpace Solutions. The side facing out to the floor is essentially a glass wall. This gives observers in the conference room a clear view of the 40-foot video wall, so they always have awareness of what’s going on in the command center. The room can also double as a crisis center during large-scale events. “Management is able to visually see what’s going on across the enterprise, so it really enhances their decision making and response," adds Marcello. “The layout really fosters that.”
New technology also essential to decision making, response
Duke Energy also has the distinction of being one of the first electric power companies in the U.S. to successfully deploy PSIM (Physical Security Information Management) technology in its security operation.
“We began looking at PSIM about three or four years ago; the primary driver was the growing number of alarms and signals that were coming into our center that required a response,” says Marcello. “We were looking for the best and most effective way to manage this increasing workload, with a particular focus on regulatory-driven requirements — because the consequence of not applying the appropriate responses can result in penalties and violations.
“From a cost perspective, we also wanted to curb the growth in manpower by providing tools to increase our security operators’ productivity and efficiency. That’s how we came to identify PSIM, and specifically NICE Situator, as a key element of our plans going forward.”
NICE Situator enhances efficiency for security operators by automatically analyzing and correlating information and alerts across security subsystems in real time. Adaptive, predefined response plans embedded in NICE Situator then guide security operators to follow specific standard operating procedures based on the indicators of what’s happening. The solution also documents incidents for investigations and compliance reporting.
Greater than threefold increase in monitoring capabilities without adding staff
Prior to implementing NICE Situator, Duke Energy’s former Security Command Center (known as the “Security Console”) was able to monitor 15 critical sites with fewer than 100 sensor points. Today, with the addition of NICE Situator, the new ESCC has been able to grow its monitoring capabilities by more than a factor of three, without adding staff. Security operators use NICE Situator to keep watch over 53 critical sites (a number anticipated to grow to 110 by year-end). Critical sites include substations, office buildings, operations centers, transmission lines departments, hydro facilities, generation facilities, etc.
Alerts that come into the Security Command Center are either security-related (e.g., door-forced-open alarms and door-held-open alarms), or of the system variety (e.g., communication failures of access-control panels and readers).
When an alert comes in, NICE Situator displays the core information the security operator needs to respond or deploy local security forces.
The alarms are overlaid on a map-based interface complete with building graphics, and supplemented by structured and easy-to-follow action plans. That allows operators to quickly pinpoint and direct response personnel to the alarm’s exact location, even if it’s on a sprawling campus or in a very large building.
But it wasn’t always that easy.
Before NICE Situator, security operators would have received general details about the alarm location, but not in a visually intuitive way.
“An alarm might have come up as ‘door number 123’ at a particular operation center, but the operator really wouldn’t have a clue where that door was because most of them would have never visited the location,” explains Marcello.
In contrast, the ability to visualize the alert on a facility’s floor plan is very helpful, Marcello says. “Some response procedures require the operator to call local personnel and notify them of the alarm. Without the visual reference, the operator can only say it’s ‘door number 123.’ The person on the other end of the line may not know where that is. But if operators can visualize alarms, they can explain ‘it’s near the computer room,’ or give other directions.”
Streamlined processes cut training requirements, accelerate response times
The ESCC has also benefited from NICE Situator’s ability to automate response procedures. Before the PSIM system was deployed, when an alarm came in, the security operator would have to manually look up the response procedure in a notebook, and that was time-consuming. Plus, there’s always potential for error, even for more senior operators who believe they know the procedures, Marcello points out.
Now, standard operating procedures (SOPs) can be stored in NICE Situator’s database and linked to specific alarm-sensor conditions. When an alarm is triggered, the right procedure is automatically displayed for the operator to follow. If the operator doesn’t acknowledge the alarm or misses a step, the system automatically escalates the alert to the next level of supervision in the ESCC.
Currently, Duke Energy has 185 standard operating procedures implemented in Situator.
“Operators see the alarms coming in, they see where the alarms are physically located, and they see the steps, the actions they’re required to take,” says Marcello. “If there are other devices available, video cameras for example, they see the on-scene video as well. The PSIM solution helps operators better assess situations and respond in the appropriate manner.”
Security operators are now able to do all their work in one integrated, automated system, and Marcello says that streamlined process has cut training requirements and accelerated response time by 50 percent. Because SOPs are maintained in the PSIM database instead of on paper, workspace clutter is eliminated, and procedural changes can be easily implemented.
According to Marcello, the PSIM also made it easier to consolidate security operations for Duke Energy and Progress Energy when the two companies merged last year.
“NICE Situator gave us a framework to automate and transition disparate operating procedures onto a common platform to facilitate consistent response actions across our entire security enterprise,” he said.
Up next: merging monitoring of regulated sites into Situator
In addition to monitoring the 53 critical sites, the ESCC monitors 35 regulated (NERC CIP) sites as well. (Essentially, NERC CIP sites are locations that house Critical Cyber Assets as defined by NERC, the North American Electric Reliability Corporation.) These sites are subjected to more stringent security requirements to protect them from threats that could otherwise affect the reliable operation of the bulk electric system.
Currently, a separate team of security operators in the ESCC oversees security for these regulated sites by monitoring alarms from two legacy Physical Access Control Systems (PACSs), remnants of the company’s premerger days.
Duke Energy plans to migrate these legacy systems over to a single, new enterprise-wide Physical Access Control System. Once this migration is complete, the new system will be integrated into Situator.
“The intent is for all sites across the Duke Energy enterprise to be monitored through Situator, regardless of how they’re classified,” says Marcello. “The strategy over the next 12 months is to bring all of our access control and video into a single platform, to integrate these various tools through the PSIM.”
One huge benefit of this approach is that monitoring of critical and regulated sites will no longer be handled by siloed groups; it will come into the center in a single prioritized work queue. That, in turn, will help distribute and balance the workload better. Any operator will be able to monitor any incoming sensor data input regardless of whether it originated from a regulated or critical site; the automated procedures will be there to guide their response.
“Situator will also enable us to increase our capacity to take on new work in the ESCC without adding headcount,” says Myers. “We’ll be able to put priorities on the alarms so that the most critical ones rise to the top.”
Myers says this is particularly important given that NERC CIP-002 version 4 will go into effect sometime in 2014. NERC CIP-002 version 4 applies new bright-line criteria to determine what constitutes Critical Assets and Critical Cyber Assets, and this will greatly increase the number of facilities that need to be protected under NERC CIP.
Compliance with NERC CIP is enforced through NERC audits, and utilities found to be in violation can be subjected to large fines. The burden is on the utility to provide documentation and proof of compliance.
“The ability to document incidents and generate reports, to integrate this function into the process, particularly on the regulated side, is especially important to us,” Marcello adds.
With NICE Situator, every single action taken during the course of incidents will be automatically captured and saved, and readily reproducible in PDF format for compliance audits. Reports can be generated automatically in a fraction of the time it would take to manually pull the information together.
Transforming security operations: the power of PSIM
Myers foresees other benefits to consolidating monitoring of regulated and nonregulated sites through Situator. “It will allow us to have one set of training for all of our operators. If you’re an operator and you come into work you can sit down in any seat in the ESCC and have the same experience. Everyone will have the same training.”
He also says that while PSIM is often equated with improving operator efficiency, it’s important not to overlook another aspect — the operator experience.
“No doubt Situator improves efficiency, but I’m also looking at it in terms of the experience for the operator. One of the big problems with security monitoring is if you’re perpetually trying to watch for something to happen, for hours and hours on end, you can get fatigued. Situator allows our operators to focus on handling real events that are brought to their attention, and then go back to a state of rest in between.”
For Myers, this human factor is just one of many examples of how organizations can put the power of PSIM to work to transform their security operations.
Myers credits Rocco “Rocky” Marcello’s forward-thinking vision for laying the foundation for Duke Energy’s successful deployment of PSIM. Marcello will be retiring from Duke Energy in October. Regie Bryant, who has been with Duke Energy for more than 28 years, will take over the project as the new director of Infrastructure Protection Services, and Brian Smith, who manages the ECSS day-to-day operations, will assist him.
“This is a huge success story for Duke Energy,” says Myers. “First and foremost, I want to recognize Rocky because I’m not sure how many people would have been able to make this happen. I have high expectations for Regie and Brian to complete this transformation into a world-class enterprise security command center.”
