I and many of my colleagues on the ASIS IT Security Council can easily take our IT knowledge for granted, and a recent experience during a video security system acceptance test brought its value home. The situation reminded me of a question that I am often asked when I assert that network monitoring tools (either commercial or open source) should be included in every networked security system installation.
Q: We are basically installing our video cameras on a dedicated LAN, with no other devices (security or otherwise) on it. Why would I bother putting network monitoring tools on that network? The video management software will report video loss from the camera.
A: Loss of video could be from a network switch problem, cable break or disconnect, power loss at the camera (PoE or mains power), or a denial of service condition due to traffic overload. Monitoring all the devices in the network (including PoE midspan devices) can save a lot of needless walking and help quickly pinpoint the source of the problem. Most integrators are not set up to quickly troubleshoot network issues.
Here is what occurred during the recent networked video system acceptance test. It is worth reading this story to get a small taste of what the ROI can be from having network tools in place. This was a small video system with 33 cameras, a video management system (VMS) server and a single touch-screen all-in-one workstation computer for the manager of the property. The touch-screen workstation was originally set up, connected to the network, and tested in the video server room. Everything was working as expected. All of the training exercise steps had been executed perfectly. Then the workstation was moved up to the second-floor conference room for the training and acceptance test.
Viewing live video was working, but it seemed jerky. Calling up events recorded by motion trigger took 15 to 30 seconds, when it should be nearly instantaneous. Video recorded continuously would not display at all, even after waiting for minutes.
One integrator’s technician suggested that it might have something to do with having powered down the cameras a little earlier. Another tech suggested restarting the video management system server. This made me nervous. A restart means that the technician does not actually know what’s wrong, and he’s hoping that it will magically be fixed by the restart. However, if the restart does work, the root cause of the problem goes undiscovered — that’s not good. Restarts are the experienced IT person’s last resort, not the first action.
All of this discussion occurred in front of the customer, and I was concerned. If cycling power to the cameras causes a major problem, then the system isn’t sufficiently robust. The customer’s Purchasing Department representative was savvy enough to know this. If the restart failed to fix the problem, the technicians would resort to calling the VMS maker for trouble-shooting support, and I would have to reschedule the training and acceptance session.
The customer’s IT technician said, “I saw the workstation working fine a week or so ago in this very conference room. And I saw it working just fine in the server room this morning.” He then asked, “Did you carry that network cable up with you? It doesn’t look like one of mine.”
“We pulled the cable from our bag here,” the integrator’s tech replied.
“Try this one,” said the customer’s tech, pulling a cable from his kit. The system worked just fine from that point on. The integrator’s tech had used a bad cable.
What Could Have Happened
Trying to view video over the bad cable was causing excessive network data packet transmission retries and high data loss between the network switch and the workstation. Had SNMP (Simple Network Monitoring Protocol) software been properly in place to monitor the video LAN, it would have been a different story. The SNMP software would have sent alerts to the technician via e-mail or text message, and he would have quickly identified the cable as being the most likely source of the problem. All would have been fine before the customer walked into the room. No 20-minute delay, no wavering of customer confidence, no calling the VMS software into question, no needing the customer’s IT tech to come to the rescue.
Now, the customer is wondering if the integrator is sufficiently qualified to install the other three larger systems that they need at other properties.
How would the training session have started had network monitoring software been in place? The integrator’s tech would have held up his smart phone, showing the customer’s IT tech the message about excessive packet retries and packet loss. “Bad cable; threw me for a minute,” is all he would need to have said. That would have been a simple demonstration of competence.
Now that nearly all of our security systems require networks, when will integrators realize that they need to adopt simple network management practices? Maybe it will take an embarrassment or two. Or a few lost projects…but I hope not.
Write to Ray about this column at [email protected]. Ray Bernard, PSP, CHS-III is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities. He is founder and publisher of The Security Minute 60-second newsletter (www.TheSecurityMinute.com). For more information about Ray Bernard and RBCS go to www.go-rbcs.com or call 949-831-6788. Mr. Bernard is also a member of the Subject Matter Expert Faculty of the Security Executive Council (www.SecurityExecutiveCouncil.com).
