To understand the state of systems and network management for electronic physical security systems, in contrast with that of IT practice, it is important to take a little historical context into account.
In the 1980s and ‘90s, networking grew, and network uptime became important. In the 2000s, networked real time information systems became the backbone of business, and network uptime became critically important. Today, it remains an absolute business necessity.
Early on, network monitoring was developed to help IT departments assure network uptime. As systems technologies advanced, monitoring advanced from simple network monitoring to advanced server and network infrastructure monitoring. In order to keep real time applications running 24/7, IT needs deep visibility into the faults, events, capacities, performance profiles and configurations of their networks and systems.
IT can’t wait for network or system failures to occur. IT has to be highly proactive, getting out in front of problems before they become disasters. And IT has to be as predictive as possible, by watching the performance indicators of equipment and applications, to keep their organization’s information systems running smoothly.
IT achieves 99.999% uptime across a mix of multi-vendor, multi-application computing infrastructure. So can security.
Over the same time period as IT was advancing its information systems technology, electronic physical security systems evolved from standalone systems to networked and integrated systems. However, physical security’s adoption of information technology lags five to ten years behind the state of information technology in business. At the same time as the security industry was still arguing over whether IP cameras would actually “catch on”, businesses were investing in high-capacity networks and quickly moving their telephone systems from analog technology to Voice-over-IP.
Today, businesses are using advanced analytics to get end-to-end insight across their entire information systems infrastructure. Meanwhile, security directors and managers still have security officers scrolling through video camera displays in an attempt to ensure system integrity. It’s not effective, because even if a camera is “on,” that does not mean it is recording as it should be. Back in the 20th century, with analog technology, officers had to visually inspect camera status because there was no other option.
Today, it’s a different story. Advanced IT monitoring and management technology can check all devices and systems 24/7. Personnel are notified instantly of a component failure, or of a near-failure condition, and have diagnostics information at their fingertips.
Yet, security departments with 21st century security technology are still applying 20th century management practices. Neither security departments nor their systems integrators have end-to-end insight across their entire security systems infrastructure.
Individual self-test health-checks do not tell you what you need to know, when you need to know it. So it can happen that video stops recording or retention periods fail to be met and nobody knows.
This is why, in spite of video management software health-checks, investigators still find critical video missing.
This needless compromise of security’s mission stems from one single fact: The security industry also lags years behind in information technology practices. Thankfully, this is a simple problem to fix: adopt the sensible practices of IT infrastructure management.
You wouldn’t buy a new car (in fact you couldn’t even find one) that doesn’t have a comprehensive computerized diagnostics system built into it, and you wouldn’t take that car to a service center that didn’t have a separate computer system to read the diagnostics data and analyze the car’s systems in real time while it’s running. That’s the state of technology today.
Why remain a decade behind, and accept a security systems infrastructure that doesn’t have IT-class systems and network management technology as part of its build-out?
Self-Test Health-Checks Only Work on a Single Vendor Basis
Prior to networking, when security systems were standalone systems, there were no shared infrastructure elements. Today, security systems are much more complex, share a common network infrastructure, and contain software and hardware components from a variety of manufacturers. Each product can check and report on what it knows about itself, but can’t report much, if anything, about the remainder of the infrastructure.
This is especially apparent with regard to video surveillance systems. If not all of a camera’s video stream data is getting through to the video server or its storage server, there can be any number of reasons for that problem that have nothing to do with the camera itself or the video server.
Today’s sophisticated applications also have a certain degree of fault-tolerance, which is why video management system software can continue to record video even when some of the video frames are not successfully transmitted. This is one reason why not all problems are visible on-screen. It takes automated technology to detect this kind of problem. Without comprehensive performance and diagnostics information for the entire infrastructure, there is too little information for troubleshooters to go on, especially for intermittent problems.
Taking a step back, one can see other factors involved in why self-test health checks fall short.
Self-test health checks remain narrowly focused because no vendor will invest in making his competitors’ products and systems run better. Additionally, each product is focused on its own functions. It’s not looking at what happens between itself and the other vendors’ products and systems that it interacts with. This is why third-party technology (i.e. something outside the physical security systems themselves) is required for infrastructure management. This is true not just for video, but for all of the security systems technology.
These are just some of the reasons why IT doesn’t rely on self-test health checks to manage its information systems technology. You shouldn’t either.
Using IT-Class Systems Management
A 21st century automated approach to security systems management makes a big difference in system and device uptime. Here are some of the differences between current common security industry practices and current IT practices, and the results of applying the IT practices. Note that there are a few industry-leading companies who already apply IT practices and make maximum use of service management tools; however, they are the rare exceptions.
Infrastructure Documentation
- Industry Practice: Most security systems are poorly documented if at all; network documentation often exists but is rarely consulted; cable labeling follows no particular standard and if done, deteriorates over time.
- IT Practice: Infrastructure documentation is produced by comprehensive automated discovery and is kept up to date; equipment and application configurations are well-documented; cabling and other physical infrastructure elements are permanently labeled according to common practice and organization standards.
- Result: Service work is accurate (no configuration guessing) and no time is wasted figuring out what connects to what; service costs are reduced and repair times are shortened.
Finding Problems
- Industry Practice: Some trouble symptoms are randomly discovered by end users.
- IT Practice: Automation finds not only symptoms but also root causes.
- Result: Instant problem detection occurs plus early warning for situations near their problem threshold.
Assessing Problems
- Industry Practice: Guesswork and past experience are used; there are limited diagnostic tools.
- IT Practice: Scientific analysis utilizes complete infrastructure knowledge and real-time status information.
- Result: Rapid root cause identification, drastically reduced troubleshooting time, and elimination of needless truck rolls for troubleshooting are achieved, resulting in much greater service capacity per technician.
Diagnostic Information Storage
- Industry Practice: What little information that is collected is stored on the system’s own server, requiring server access for problem troubleshooting; data is lost if the system dies, data is inaccessible if the application crashes or the intelligent device is offline or had died.
- IT Practice: Diagnostics information is centralized outside of the monitored systems, and is also backed up in the cloud (a growing trend).
- Result: Direct access to the critical system is not required for diagnosis; if the system or device fails the troubleshooting information is still available.
Corrective Action Plans
- Industry Practice: Fix-it actions are performed ad-hoc based upon end-user or systems integrator experience.
- IT Practice: Corrective and preventive action planning is based on automated root cause analysis supported by a comprehensive technology knowledge base.
- Result: Corrective actions actually are correct and thus permanent; the technology knowledge base is enhanced; preventive measures improve uptime even more.
Informing the Team
- Industry Practice: Ad hoc phone calls, emails and text messages are used; notifications to stakeholders are inconsistent and insufficiently informative; high level access passwords are shared insecurely for systems and devices (plus integrators commonly use master passwords used across a wide customer base); diagnostic information with sensitive data is often over-shared and ends up residing in poorly secured outside computers.
- IT Practice: Automated stakeholder notifications are performed instantly with message content appropriate to the stakeholder’s role; alerts (failure or critical problem), warnings (failure likely) and advisories (performance below par) with appropriate diagnostic information enable support teams to be maximally proactive and to correctly assign and prioritize service tasks; automated service management tracks workflows and escalates stale task completions.
- Result: Service work dropped balls are eliminated; system downtime is minimized; team members are optimally utilized; customers are kept fully informed; and service is highly efficient and very cost-effective.
Collaboration Technology
- Industry Practice: Telephone conference calls and desktop sharing technologies are used; remote access to live systems is used and sometimes shared (such as LogMeIn or TeamViewer) and is usually not logged or tracked (end-user customer may have no knowledge); industry practice around remote access security is very poor; collaboration is often sequential (one vendor or service provider at a time); lacking hard diagnostics data, conference calls often result in finger-pointing.
- IT Practice: Purpose-appropriate collaboration technology is used to securely share diagnostics information; remote access to live systems is limited to authorized technicians making corrections; all systems stakeholders share the full scope of diagnostics data (technology vendors, technology service providers, in-house network personnel, and end user).
- Result: All parties are fully informed with good diagnostics data; finger-pointing is eliminated; collaboration time is shortened for all parties; and no diagnostic information resides outside the customer organization’s data repository.
Self-test health checks don’t provide the kinds of results listed above. There is real Return-On-Investment in the IT practices, which that security industry self-test health checks just can’t deliver.
Management, compliance officers, financial stakeholders and those responsible for critical information systems operations rely on scientific proof (auditable proof) of the performance of IT devices and systems. Physical security systems stakeholders have the same right to get proof of performance like what the organization gets for its IT systems. Qualitative opinions and guesswork from vendors, security staff or integrator service technicians are not sufficient.
It is long past time to elevate security systems management to a level of practice that is appropriate for today’s 21st century information-technology-based security systems. IT has already proven the cost and system uptime benefits. Businesses expect, and most mandate, that their significant technology investments be properly managed and well-performing.
It is time to start making the business case for IT-class systems management for physical security systems.
About the Author: Ray Bernard, PSP, CHS-III is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities. Ray is an active member of the ASIS Physical Security Council and the IT Security Council. For more information about Ray Bernard and RBCS go to www.go-rbcs.com or call 949-831-6788. Follow Ray on Twitter: @RayBernardRBCS.
