For the past two decades, the security industry has struggled with the adoption of information technology. Technology advancement continues to accelerate, as do end-user customer expectations of security system technical capabilities.
There are many articles about security technology that discuss supporting user needs and expectations; but what about the needs of companies who deploy and maintain the technology? What are their expectations? Talking with security integrators at ISC West, I found two primary concerns for advancing security technology: cybersecurity and manageability at scale.
These are both major points of weakness for many security system deployments – especially those with high camera counts. The two concerns are interrelated, because if you cannot easily update large deployments, it is impossible to keep them cyber-secure.
Integrators have seen the embarrassment caused to security managers and directors who have had to explain to IT personnel or higher-ups that the camera system that cost hundreds of thousands to millions of dollars does not meet IT department requirements, cannot be secured like the organization’s other computer and network systems are, and has huge labor costs involved in keeping the camera firmware up to date.
Thankfully for both integrators and end-users, new tools are emerging to address these two critical issues.
Cybersecurity
Integrators are generally not up to date on common cybersecurity practices. For example, the six most important of the top 20 cybersecurity controls from the Center for Internet Security (www.cisecurity.org/controls) are:
- Inventory and Control of Hardware Assets
- Inventory and Control of Software Assets
- Continuous Vulnerability Management
- Controlled Use of Administrative Privileges
- Secure Configuration for Hardware and Software on Mobile Devices, Laptops, Workstations and Servers
- Maintenance, Monitoring, and Analysis of Audit Logs
Secure configuration includes keeping devices at the most current level of firmware. All six of the basic controls apply to electronic security systems. Until now, there have not been tools to automate and standardize the steps required. Security system and device hardening guides recommend these steps, but they always had to be performed as a fully manual process, and so rarely were done well – if at all.
Manageability at Scale
“Service assurance” refers to keeping devices and systems “in-service” (i.e. online and functioning). It is the application of policies, processes and tools to rapidly, efficiently and cost-effectively identify, isolate, troubleshoot and repair problems with devices, computers and networks that impact security system performance. Cybersecurity vulnerabilities are now among the system problems that, once discovered, require immediate correction. Manual processes are not time-feasible or cost-feasible for maintaining system and device cybersecurity and full performance.
Decades ago, the IT industry realized that it requires automated software tools to monitor and manage large scale information systems. This aspect of information technology has not been embraced by security industry manufacturers – until now.
Two Breakthroughs at ISC West
Historically, the greatest cybersecurity vulnerability for networked security systems has been factory-default passwords, and the next greatest vulnerability has been the infeasibility of managing large scale camera deployments. Managing firmware updates for 2,000 cameras was simply not feasible – and that’s why, over the past two years, 1.5 million video cameras and recorders have been compromised with botnet malware infections.
Two products – both automated software tools – were introduced at ISC West 2018 that for the first time bring manageability to large scale camera deployments. Using these tools, it is possible to bring security video surveillance systems into compliance with typical corporate IT cybersecurity policies and practices.
The products are looked at separately because while the scope of their functionality overlaps, they are not identical. Both significantly reduce the total cost of ownership for surveillance systems; and both provide automated firmware management, which is what makes them truly breakthrough products.
Axis Device Manager
The AXIS Device Manager (ADM) from Axis Communications (www.securityinfowatch.com/12397387) is an on-premise client-server software tool for managing up to 2,000 Axis cameras and other Axis devices (such as loudspeakers) at a single site. There is no limit to the number of sites at which the tool can be deployed. The software is installed on a site server that has network access to the Axis devices. The ADM client software can connect to multiple ADM server instances.
The product features multiple functions, including:
- Cybersecurity: A network of thousands of Axis cameras can be automatically hardened per the Axis hardening guide and kept up to date with automatic firmware updates. ADM enables centralized IP address management, account, password and digital certificate management for HTTPS and 802.1x certificates.
- Documentation: Device Manager automatically scans the network and locates all online Axis devices, including audio and access control devices, creating a database of devices and their configurations.
- Backup and restore: Most camera configuration settings can be backed up for later quick restoration. Restore points are generated automatically for all devices that ADM has discovered (manual restore points can also be created). A backup template can be set up for a group of cameras. Restore points enable users to compare the current device configuration to one contained in a restore point. By default, restore points are created every night for automatically discovered devices.
- Camera firmware manageability: Cameras can be grouped so firmware update actions can be performed by group, with an option to perform group updates in sequence or in parallel. Updating cameras in parallel speeds overall update time. There is an option to stop a group’s update sequence if the update fails on one of the cameras. Remote access to multiple ADM instances (multiple sites) can be set up to enable central management, one site at a time.
- Camera password management: ADM provides camera password management, a near-impossible task prior to its arrival. The most highly exploited cyber vulnerability is the use of default or stolen passwords for unauthorized access. Axis provides detailed guidance on the management of camera passwords in its ADM materials.
- Deployment: Settings from one configured camera can be partially or completely copied to multiple other cameras.
- Auditability: ADM maintains an activity log with sort and search functions showing performed actions and changed device status.
- Total cost of ownership: The manual labor involved in deploying and maintaining cameras is significantly reduced.
Viakoo Camera Firmware Update Manager and Password Checker
Camera Firmware Update Manager (CFUM) and Password Checker are two of the many capabilities provided by the cloud-based Viakoo service assurance solution. CFUM (www.securityinfowatch.com/12400265) supports multiple brands of cameras. A small low-overhead software agent resides on each recording server or NVR, providing integration with the VMS or NVR software and the cameras and camera network’s devices, as well as outbound-only data communication to the Viakoo cloud (no video streams or video scene metadata).
- Cybersecurity: Viakoo scans multiple networks to identify all the devices involved in originating, transmitting, and storing camera video streams, and builds a digital model of the entire networked security system for any size deployment across multiple geographic locations. It integrates with major VMS and NVR products, associates each camera with its recording device, and identifies the video stream path through all the network and video system devices that it traverses.
- Password checking: Password Checker checks cameras and VMS for factory default and easily guessable passwords, providing a report of cameras whose passwords should be changed.
- Documentation: The system builds a navigable and printable model of the entire electronic physical security systems infrastructure, including each camera’s IP address, hardware address, make, model, name or label, location information and software version of its recorder, via integration with the VMS or NVR software, and reports on results of password checking.
- Manageability: CFUM enables cameras to be grouped so firmware update actions can be performed by group, with an option to perform group updates in sequence or in parallel. Groups can be updated in parallel, to speed the overall update time. Additionally, update “jobs” are defined and can be immediately run or scheduled, as it may be ideal to have firmware updates deployed at non-peak activity times for camera fields of view. Where an issue has surfaced regarding a firmware update that has already been deployed, a “reversion” update job can be scheduled to roll back to the previously used firmware version.
- Auditability: All CFUM and Password Checker actions are logged.
- Total cost of ownership: The manual labor involved in deploying and maintaining cameras is significantly reduced.
How to Deploy Firmware Updates
Performing managed services does not mean doing ad hoc preventive maintenance and update work. It means documenting, planning and tracking service results in an auditable manner – especially work that has a cybersecurity impact.
Whether an integrator or an end-user, there are four overall steps for installing and managing camera firmware updates, and each one involves several considerations.
Step 1: Inventory the video surveillance system infrastructure: In the camera inventory, it is important to match up the camera with the version of VMS software and the versions of any third-party analytics that are being run on the camera or on a server. The VMS software and analytics software must be compatible with the version of camera firmware.
Good firmware update management includes the creation of camera make, model and firmware configurations and testing them with the related VMS and analytics software. Once tested, the make/model/firmware/VMS/analytics combinations can be approved for rollout.
Step 2: Retrieve and test firmware update files: Once the firmware has been retrieved from the vendor and tested with the various make/model/firmware combinations, it should be digitally signed and safely stored. Ideally, the following process should be followed per-camera:
- Stop recording for camera under test.
- Perform the firmware upgrade.
- If the upgrade is unsuccessful reboot the camera and restore the previous firmware.
- Restart video recording for camera.
The reason for stopping the recording for the camera update is that VMS systems react differently to the camera going offline for the camera update if recording is in progress, with some VMS software incorrectly reporting errors or failing to automatically restart the recording. A test of the VMS software’s reactions to a firmware update can be done to determine if the VMS behavior is acceptable if the recording is not stopped prior to the update and then restarted. That is not necessary with Viakoo’s CFUM, because it automatically performs those steps via integration with the VMS.
Track the time that it takes to perform the update for each make/model/firmware combination, including the time for any manual steps involved for each camera. For some cameras, this can be a 10-minute process. Noting the time involved enables video monitoring personnel to be alerted that cameras will be taken offline for an approximate range of time according to the update schedule.
Step 3: Perform scheduled firmware update rollout: Both ADM and CFUM enable the “batching” of firmware update actions; however, careful consideration should be given to which cameras are selected for parallel updating. For example, in a casino, updating the cameras for each game table should be performed in sequence, so that a table only loses one camera’s coverage at a time. The same is true for cameras covering regulated storage areas that have video monitoring for DEA, CTPAT, PCI and other regulations and voluntary programs. Notify monitoring personnel at the start and completion of the camera update jobs affecting them, preferably, using automated notifications set up in the firmware update tool.
Step 4: Monitor firmware compliance and success status: Collect new firmware release data and update the camera inventory to identify newly non-compliant (i.e. non-updated) cameras. Schedule “firmware reversion jobs” for a firmware release that turns out to be troublesome and requires fallback, applying it only to applicable firmware/make/model/VMS/Analytics combinations. Provide compliance reports for auditors. Do not omit this important step for trouble issues found: create trouble reports for discovered issues, and send them to the related camera, VMS, NVR, and analytics vendors.
Ray Bernard, PSP CHS-III, is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities (www.go-rbcs.com). He is the author of the Elsevier book Security Technology Convergence Insights available on Amazon. Mr. Bernard is a Subject Matter Expert Faculty of the Security Executive Council (SEC) and an active member of the ASIS International member councils for Physical Security and IT Security.
