What we can learn about building IT/OT redundancy from modern emergency operations centers

Ensuring that all technologies, utilities and building facilities are continuously operational is essential to almost anyone who owns and operates a building or campus. However, the level of redundancy and operational/security resiliency required will differ depending on the functional requirements.

The best examples of redundancy and resiliency at the far end of the spectrum are modern emergency operations centers (EOCs). Today, EOCs are designed and built to function fully during significant natural and human disasters.

Let’s learn how current redundancy and resiliency techniques are applied to emergency operations centers and what capabilities or benefits might apply to other buildings or campuses that wish to maintain elevated IT and OT operations levels.

The Purpose of Redundancy and Resiliency within Emergency Operations Centers

While emergency operations centers primarily serve as around-the-clock E-911 contact and dispatch locations, the facilities are often built to serve multiple governmental agency purposes and use cases in times of crisis. For example, in EOCs deployed in areas where hurricanes, tornados, flooding, wildfires and other natural disasters occur, the facilities and associated IT/OT systems are built to withstand these calamities and remain fully operational.

Because EOCs are essential to preparedness and rapid response, they serve as a critical meeting location for decision-makers, local and federal agency representatives, incident management personnel and other subject matter experts. Communication is crucial in times of emergency, and having a reliable location for these experts to meet and coordinate a plan of action can significantly reduce the time that plan is set in motion.

In most cases, EOCs house several essential components, facilities and teams beyond the E-911 contact center to help coordinate emergency response and establish reliable communications with the public in life-saving situations. This includes hosting critical IT and communications infrastructure, disaster-response meeting facilities, and plenty of open office space that external agencies and experts can use in an emergency.

Interestingly, EOC architects may only design a portion of the overall EOC building with redundant power, lighting, HVAC and other required redundant features. This is done as a cost-savings measure to provide redundancy to only the locations where redundancy and resilience is necessity.

Building EOCs That Withstand Natural Disasters

EOC-designated buildings near coastal waters that often experience natural disasters such as hurricanes must be built to withstand intense wind and flooding while maintaining a 100% operational state for extended periods. However, buildings are only useful in these digital times with the necessary utility, IT and OT components and services that allow emergency teams to communicate and coordinate effectively.

From an operational technology perspective, modern EOCs are outfitted with fully redundant power, sanitation and HVAC systems that ensure ongoing operations during a disaster. Data centers are commonly built using standards-based redundancy models. Depending on the facility, these data centers operate using N+1, N+2, 2N, and 2N+1 redundancy models. From a high-level perspective, here is the breakdown of the levels of redundancy each model provides:

• N+1 redundancy level: Delivers the minimum necessary power and cooling to a facility with the necessary power backup. Additionally, an extra single battery backup or generator system is included that can be used in the event that where a primary power backup system fails. For example, if an EOC has five battery backup systems allocated to communications equipment, N+1 mandates that six battery backup units be installed if one of them were to fail.
• N+2 redundancy level: In many cases, N+1 isn’t sufficient for EOCs; thus, an N+2 redundancy strategy is put in place. This provides two additional power backup systems to be placed online, protecting against the failure of two operating power backup systems.
• 2N redundancy level: This level creates a complete copy of power backup (battery and/or generator) units that can be accessed if the primary power backup system fails.
• 2N+1 redundancy level: Adding one additional layer of power redundancy not only provides a complete mirror image of the primary backup power system, but also adds a single unit of extra protection if the mirror system incurs a failure on one of its units.

Data BCDR and Security Protections Integrated into EOCs

Protecting applications, digital services and data from the threat of outages and data loss is paramount within emergency operations centers. Preventative measures of EOCs must adhere to strict business continuity and disaster recovery (BCDR) standards. This includes regularly scheduled local and offsite backups and regularly scheduled data recovery tests.

IT security tools commonly implemented in enterprise-grade data centers, campus LANs and clouds are also required to ensure that digital services remain online and free from compromise. This includes modern data security monitoring and threat prevention tools like firewalls, intrusion prevention systems (IPS) and data loss prevention (DLP) systems.

Network segmentation within EOCs is also critical. EOCs use designated “hardened” areas that add extra precautions and tools to better protect against service disruptions of essential communications, IT and OT systems that are required during a disaster. Logical micro-segmentation allows the entire EOC to connect to a single network deployed within the building. However, granular access control configurations are assigned to these mission-critical systems that only allow these applications and services to communicate with predetermined servers, storage and endpoints. In the event of a security compromise, micro-segmentation prevents malicious actors from moving laterally within the rest of the network.

EOC Redundancies That Can (And Should) Be Integrated Into Any Commercial Building

It’s safe to say that most commercial buildings do not have to be built to specifications that withstand hurricane-force winds or missile strikes. It’s also safe to say that much of the higher facility and power redundancy levels are overkill. However, keep in mind that as the use of in-building technologies grows from both an IT and OT perspective, it’s possible that some of the redundancies and resiliency measures used within EOCs will begin to trickle into standard commercial facilities.

Those looking to integrate high levels of redundancy into their buildings can learn significantly from the technologies and techniques used to build emergency operations centers. This is especially true for buildings outfitted with smart building technologies that control IT/OT systems and network infrastructure that support physical security, health and tenant quality of life IoT technologies. These areas must be hardened with redundancy, security and resiliency enhancements to ensure operation around the clock.

Simply put, the types of data security and power redundancy enhancements found in EOCs are quickly becoming vital as in-building technologies shift away from being a “nice to have” to a “must have.”