Security Innovation Awards Silver Medal: Leadership and innovation merge to secure Houston-based hospital system

When I was interviewing to become the system security executive at Memorial Hermann Health System I got the sense that the systems and procedures were in need of significant attention and remediation. It wasn't until my first days on the job before I was able to recognize just how daunting this task would be.

Memorial Hermann is a network of 13 hospitals located in and around Houston, Texas. Security professionals know that one stand-alone hospital is a complex environment requiring technical, managerial and financial savvy unlike any other vertical.  Now multiply that level of complexity by 13 and put it all under one umbrella. Covering over 20 million square feet, with 1,762 doors, 1,600 cameras, 30,000 cardholders and 5 million visitors annually, this project is among the largest on the Software House C•Cure 9000 platform.

As we began analyzing the organization's security infrastructure I found dozens of offline cameras, inconsistencies in database management, unused hardware components collecting dust, and a security culture as disparate as the independently operating access control systems. We needed a significant investment of financial and human resources to right the ship.

Each site (except one) was operating a local version of Software House C•Cure 800, so this is where we started. I contacted Software House and explained my dilemma. I felt that the best solution was to centralize the security systems, but in order to accomplish this we needed an integrator that understood our needs and could deliver on our expectations.

As I interviewed C•Cure dealers, Software House introduced me to Scott Welborn from Tech Systems, Inc., one of Software House's enterprise-level dealers. Scott listened intently to my concerns, needs and vision. Usually, once this kind of project would go to bid to the few Software House C•Cure dealers in the area, bids would be proposed and typically, decisions would be made in favor of the lowest-cost, unqualified provider.

Scott, however, shared the details of Tech Systems' “For Our Customers' Ultimate Satisfaction” (FOCUS) program with us. We found that Memorial Hermann was able to fill in the gaps of our security operations by allowing Tech Systems to manage ongoing security procedures. The Tech Systems proposal was priced right and the value of the FOCUS program created even more value than the items that were included in the bid.

We were impressed by their attention to customer service, an excellent preventative maintenance package, a strong internal technological knowledge among the staff and proven project management methodologies.

Because this conversion was a long-term plan it required a long-term partnership. It is never good to create an adversarial relationship with vendors. They need to perform, they need to be motivated and they need to value Memorial Hermann as a customer. It's not all about price, it's about value. Tech Systems offered me the most cost-effective and results-oriented proposal. Leveraging their expertise, experience and ongoing professional services we could save money over time and have a security program that is fluid and effective today, tomorrow and a year from now.

The Plan

As a non-profit organization, Memorial Hermann had a budget and a limitation on what we could spend. The key was to get the most value for the dollar and to formulate a return-on-investment that our C-level executives could recognize.

So I got in front of 13 C-level officers and presented our plan.

The plan was to migrate to the Software House C•Cure 9000 platform, with all sites centrally managed at our headquarters location. Integrating the Hugs infant-protection product; six different video surveillance technologies that includes American Dynamics, Pelco and Intellex; along with five audio products would be no easy task. This strategy was developed after several months of in-depth discussion and collaboration with Tech Systems, Software House, Memorial Hermann IT and security staff.

The plan resolved the various issues of disparity, operational inconsistency and offered a long-term savings on licensing fees. The plan also presented new challenges, both technically and logistically.

We formalized three main objectives. They were to:

• Obtain buy-in from various organizational business units.
• Utilize existing network infrastructure for fault tolerance and resiliency.
• Maintain 24/7 operational effectiveness throughout the conversion process.

After identifying the objectives, the team conceptualized the solutions road map, which included:

• Upgrading the access control software to support centralized management across multiple locations, which meant replacing the antiquated access control product previously installed at the Texas Medical Center with C•Cure 9000.
• Install Hugs and integrate it with the C•Cure 9000 system.
• Rebuild command center at Memorial City Hospital.
  • Include new top-of-the-line computers and associated hardware and software components for a scalable, long-term video management system.
  • Include a new video wall and new desktop monitors.
• Upgrade command centers at other locations.
• Rebadge all staff members, which included updating databases, deleting unnecessary entries and editing the partitioning for every badge-holder.

Unfortunately it's common for security directors to feel we can't afford the solutions we need or that we will not be able to get approval for our plans. But we were able to get the plan approved by following several strategic courses.

Among the most important was working to eliminate the turf battle by reminding people our intentions were aligned with the needs of the organization and that the proposed methods supported our initiative. We also spent time educating the decision-makers on the pros and cons of the current system implementation and on how our proposed design would enhance the current level of security and situational awareness.

We highlighted the cost savings, factored in new efficiencies and developed a long-term budget, which created a strong financial argument. And finally we demonstrated efficiency and effective utilization of manpower.

We were successful and obtained the necessary approval to execute a 3-year FOCUS Agreement at $650,000/year including systems remediation, guaranteed response and basic system administration.

Deployment

James Wright and Daniel Hickman were the leading Tech Systems engineers assigned to our conversion project.  It didn't take long for the mistakes of the past to emerge.

First, we identified what technology was there and determined what was working and what wasn't. Then we dug into the individual systems and found a lack of standardization for data input and management. Together we developed a congruent method for the data input and maintenance.

Upgrading a legacy security system comes with many unknowns. The performance and wiring for each of the access control and video surveillance components needed to be tested and assessed to allow Tech Systems to perform the necessary updates and replacements.

We also discovered that four of the 13 locations were not on the Memorial Hermann corporate WAN. This presented a major challenge to get the IT department on board with expanding the corporate network to accommodate our request to converge the security system.

The IT staff was initially reluctant, but cooperated as every department became aware of the need for this transition. Greg Trautman and Carol Hawthorn of Memorial Hermann IT staff were integral to this process and continue to work closely with Tech Systems for the ongoing conversion process and necessary maintenance.

The architectural challenge we faced was adherence to Memorial Hermann's policy around virtualization. We took an innovative approach using a Microsoft SQL Server in a Geo-Cluster configuration. This provides a complete failover solution and in-house redundancy. The benefit is that system failures can be recovered very quickly, sometimes seamlessly. The Memorial Hermann IT staff could better control and manage the resources they allocate to the security system by monitoring the workload on the database.

In a previous attempt to upgrade the system, Memorial Hermann encountered a tremendous amount of issues resulting from an offline system and unlocked doors. It was extremely important for the conversion to work properly the first time without leaving Memorial Hermann vulnerable.

The team soon developed a migration path that converted each site one component at a time. To accomplish this, Tech Systems used a fail-safe mockup of the new system and operated it for a full week prior to deactivating the old system. Once the testing period was complete the original C•Cure 800 machines ran alongside the new C•Cure 9000 platform, as scripts extracted the data and then imported the data into the new platform. The team then tested each system, one component at a time.

The collaboration between our team, Tech Systems and Software House was critical. The conversion is successful and the deployment has been on track as a result of our close partnership and ongoing communication efforts. The frequency of our conference call schedule symbolizes the progress we've made in the project. In the beginning we would hold daily conference calls to keep everything on target. As we progressed we adjusted to weekly conference calls. Now we are far enough along that we are preparing an adjustment to monthly calls.

As the focus changed from converting the old system to operating the new system, Tech Systems, Software House and Memorial Hermann staff collaborated to establish a set of go-forward standard operating procedures for managing credentials and continuous updates.

Adjustments to the System

Badging: Memorial Hermann recently re-branded the organization which created one large unexpected project -- rebadging for approximately 30,000 badge-holders. This process could have easily become a logistical nightmare with so many partitions, schedules and the vast number of badges that needed to be re-printed. I contacted some other vendors to get pricing for the cards and then came to the realization that Tech Systems could incorporate this into the ongoing service agreement.

Tech Systems works closely and directly with the Memorial Hermann HR, IT and engineering departments to ensure the database is continuously updated for accuracy, provisioning and de-provisioning of credentials.

An integration between Memorial Hermann's iTRUST HR database and C•Cure 9000 has helped automate the database updates and has created new efficiencies and improved the integrity of our data.

Hugs Management: There are many different perspectives and stakeholders in every department at every location of Memorial Hermann, and like any change, reorganizing the way the Hugs system was managed created some new resistance. 

The nurses that were in charge of managing the Hugs system were deeply passionate about the way things were operating. So coming in and offering to manage it wasn't quite like offering to carry in a bag of groceries. They had spent their own time learning how to use it and were proud of their abilities to do so. They also have emotional relationships with the very infants we are monitoring and felt uncomfortable delegating any of that process.

To provide the security department with oversight capabilities while allowing the nursing staff to continue caring for the patients, we cross-trained our staff and leveraged the nurses' expertise. This allowed us to more intelligently integrate our security procedures into their baby-birthing procedures.

Conclusion

While much of the hardware and infrastructure remains from the days before I joined Memorial Hermann, and before Tech Systems took over as Memorial Hermann's security integrator, the conversion project has been in process for over a year and will continue for several more months. The collaboration and true partnership between every leg of this 3-legged stool has made for a very smooth transition. This team has demonstrated true professionalism - and the patients, staff and visitors of the Memorial Hermann organization will reap the rewards of our hard work for years to come. We held Tech Systems to high standards with performance reports and benchmarking against national benchmarks and internal benchmarks.

Here's a bit of advice for my peers out there based on what I learned throughout this project -- get educated and maintain your education. Take advantage of the training available to you by your vendors, your partners and your associations. The more you know about the technology and the success of others the more you can apply that knowledge to your job. Learn to lead and to communicate effectively with not only your department, but with all departments. Learn to calculate return-on-investment so you can get the funding you need to be effective. Be open-minded to new ideas and solutions, and depend on healthy relationships with your providers. When you are able to do that, your department and the organization you work for will be much better off.

About the Author:

Joseph Bellino is the System Security Executive at Memorial Hermann Healthcare System in Houston, Texas.

Memorial Hermann Project Solution Providers:

Access Control

• Software House CCURE 9000
• GE Diamond II – TMC Hospital

Video Recorders

• American Dynamics
• Pelco
• Salient
• Bosch

Video Cameras

• American Dynamics
• Pelco
• Axis
• Bosch
• Sony

Intercoms

• Aiphone
• Viking
• Talk-A-Phone
• Code Blue
• TOA