Keeping Your Network Video System Up and Running

Short of catastrophic disaster, most networks today deliver between 99.99 percent and 99.9999 percent availability — from less than one hour of downtime a year to less than one minute of downtime a year. But even the most robust network needs downtime for component maintenance and system upgrades. The difference with video surveillance systems is that downtime can jeopardize public safety and compromise the security of assets, operations and facilities. While infinite redundancy would certainly solve the problem, such a strategy would be cost-prohibitive and impractical.

How do you address the challenge of designing your video surveillance system to minimize the risks of downtime? Different strategies apply for each system element — the network, the servers hosting the video management software, storage and the network cameras. Depending on the component, there are a number of best-practices options for maximizing availability.

The Network
Hot swaps and redundant power supplies on switches and other failover technologies are good options for the most critical components of the WAN or LAN. You can monitor these devices using tools ranging from simple bandwidth monitoring and generic Simple Network Management Protocol (SNMP) applications, to highly-sophisticated, detailed network traffic and packet analysis systems including custom Management Information Base (MIB-2) support. 

You should install cabling, especially in multi-campus or exterior mounted nodes, with lightning protection and with proper termination. The main drop, for example, requires both a patch panel termination and a jack termination. For added protection, never directly terminate the cable between the switch and network camera. For easier maintenance, make sure your service loops are at least one foot at the network camera termination point.

Since the wide area network (WAN) supports mission-critical surveillance, you should design the core as a mesh network to provide multiple alternate paths in the event of a single path’s failure. A Multi-Protocol Label Switching–Virtual Private Network (MPLS-VPN), for example, delivers performance and privacy like a frame relay, combined with the flexibility and cost advantages of packet-based IP networks, like the Internet.

Plan and test any network changes prior to installation. Run patches and upgrades in the lab or on test equipment to ensure stability before introducing them to the production environment. Best practice is to always lag behind the latest service revision releases to prevent new issues from arising. When you perform maintenance, do so in localized phases so as not to create problems system-wide. With today’s robust technology, regular maintenance requirements are minimal but they are still essential for the continued health of the network.

Servers Hosting Video Management System
Many video management applications have built-in failover server capability for its server clusters. If the primary server detects a failure in one server, another identically configured server automatically takes over all the recordings of the cameras originally assigned to the failed server. It typically takes only seconds to switch servers and requires no operator/administrator intervention. This high level of failure protection is commonly used in mid-tier and enterprise video surveillance installations.

For the highest level of protection, you can send video from the network cameras to two different servers in separate locations simultaneously. Alternatively, the backup can be sent at a pre-determined time using a commonly supported archiving feature in the video management software. If you use WAN links during operating hours, you can back up video after hours to limit any disruption to other network applications using the WAN. These offsite servers can, in turn, be equipped with RAID, work in clusters or replicate its data with servers even further away.

Additional safeguards include component redundancy such as dual network interface cards and redundant hot swap power supplies. Network and database monitoring tools such as SNMP running on a network management system ensure data integrity and help maintain server health. Typically only the most mission-critical systems use health monitoring even in the largest of enterprises. Much like archived video, system support is mainly reactive to changing conditions, even if manufacturers provide tools to enable proactive management.

Storage
Like the server component, system storage can benefit from a redundant array of independent drives (RAID), allowing users to easily replace drives in case of failure. Arranging standard, off-the-shelf hard drives so that the operating system sees them as one large logical hard disk increases storage throughput as well as reliability.

There are different levels of RAID — from minimal spares to a full “hot swappable” mirrored solution where there is no disruption to the operation of the system and no loss of data in the event of a hard disk failure.

The two most common RAID levels are:

• RAID-1 (disk mirroring): Information on one disk is duplicated onto one or more disks. This increases the reliability but also increases costs and may reduce performance as data needs to be written on two disks at once.
• RAID-5 (striping with parity): Data and parity are spread over three or more disks and require at least three disks in the array. Read performance is the same as for a single disk; write performance, on the other hand, can be lower as data needs to be written on two disks. RAID-5 can tolerate a single disk failure and still recover all data. Additionally, the disks can be made hot swappable. RAID-5 has become popular because it provides redundancy and maximizes disk space.

Network Cameras
Today’s network cameras include a built-in watchdog that automatically restarts them whenever service is interrupted. But the network camera technology continues to evolve. When you upgrade camera firmware, do the upgrades one unit at a time to avoid creating major blind spots in the surveillance network which would occur if a large batch of cameras were pulled offline at the same time. You can schedule upgrades automatically as a function of the video management software or use vendor-specific tools, such as Axis Camera Management which allows an administrator to control batch upgrades.

Also remember to periodically check that cameras are not redirected, pulled out of focus or covered by post-installation changes to the environment. Many network cameras have the built-in intelligence to automatically alert operators if they have been tampered with.

Power
All electronics require power and sometimes that is the weakest link in the chain. Providing hot swappable power supplies for the servers and switches is one possibility, but if massive power outages are a concern, consider deploying an Uninterruptible Power Supply (UPS) system. Network video systems that use Power over Ethernet, however, have the advantage that the complete surveillance system, including the network cameras, will remain operational for a period of time in a power outage.

What is Acceptable Downtime?
No system has 100-percent availability. In an analog system with VCR recording, for example, the tape needs to be changed every eight hours. During this swap out — albeit for a few brief minutes — no video is being recorded. Statistically, such a set-up delivers an availability of around 99 percent, which is considered very low in the IT industry. Proprietary DVR-based systems also have comparably low availability, often lower than the regular PC servers on which they are based.

Weigh your options. Network-based video surveillance requires a maintenance strategy the same as other network systems. Network cameras need patches and updates just like routers do to improve system functionality and security. Video archives and video management applications need updates the same as database servers and financial record applications.
Circumstances and environment dictate the acceptable downtime window for any particular organization. Educational institutions, for example, would have greater tolerances for scheduling maintenance after-hours. Industrial companies could coordinate their video surveillance system downtime with other system/machinery downtime. Even retailers would have opportunities for downtime after business hours. On the other hand, given their extended hours of operation, the transportation and medical industries face much tighter constraints on when large system maintenance can be performed.

What technologies you choose to deploy to minimize downtime for your video surveillance system will depend on the criticality of coverage and the budget at your disposal to harden the network video system.

Fredrik Nilsson is general manager of Axis Communications, a provider of IP-based network video solutions that include network cameras and servers for surveillance. This story is part of Mr. Nilsson’s “Eye on Video” series appearing in ST&D and on SecurityInfoWatch.com and IPSecurityWatch.com.