Columbus Children's Hospital plans for disaster recovery for times when it's not all fun and games

While most business professionals responsible for disaster recovery take their work very seriously, Becky Crackel views her position as nothing more than "child's play." It's not that Crackel is less dedicated than her counterparts; in fact, she is uniquely motivated to develop and to implement a successful disaster recovery plan since her actions affect the lives of hundreds of sick infants and children.

Crackel, the disaster recovery program coordinator at Columbus Children's Hospital, keeps Play-Doh and other toys at her desk (like most individuals working in the hospital's information services department). However, instead of providing entertainment, the toys serve as a reminder of the information services (IS) department's mission or, as Crackel puts it, "the reason why we come to work every day." It's a humbling assignment, but Crackel is confident that the department is up to the task.

Yet it's not all fun and games for Crackel and her peers. Since disaster recovery is a top initiative for the IS department, they stay very busy testing and managing more than 450 software applications, with more applications added regularly. The hospital has a mainframe, 240 servers and 3800 computer workstations throughout the organization. All of those systems and 100 IS staff are dedicated to supporting the 323 inpatient beds, over 14,000 surgeries annually, over a half million outpatient visits annually, disease research, hospice and home care, behavioral health, child advocacy, and other Children's services.

Historically, the hospital has proactively integrated cutting-edge technology into daily operations resulting in improved efficiency, speed, accessibility, and accuracy. Such technologies, for instance, now provide error-checking for the healthcare professionals at Children's Hospital when they administer drug dosages in fractions to small infants. Other innovative applications include electronic patient charting, physician order entry, and film-less radiology information systems.

Columbus Children's Hospital created Crackel's position last year in response to the increased reliance of hospital functions on information technology. The hospital's top management made disaster recovery a priority throughout the organization. Crackel explains that "the executive staff is very receptive and accessible. They are 100 percent behind me." Additionally, the hospital has demonstrated its commitment by investing more than $2.5 million over the past two years.

"Everyone works constructively with me to implement disaster recovery goals, because our culture is to do what's best for the children and families we serve," says Crackel.

Crackel's approach to creating a disaster recovery plan was to hire Disaster Recovery Coach Dick Regentin from Regentin Associates Inc. in Akron, Ohio. Regentin, a disaster recovery industry veteran, introduced a methodology that helped to prioritize the hospital's hundreds of systems. After studying industry best practices, conducting gap analyses and refining internal priorities, the department initiated a three-tier disaster recovery strategy.

The resulting tier system divides systems into three groups: Tier 1 are the most critical systems, about 70 servers, that would disrupt patient care and must be functioning again within 48 hours or less; Tier 2 systems should be up within 72 hours; and Tier 3 systems should be operational within five days or more.

Once the disaster recovery strategy had been formulated, Crackel set a realistic implementation plan for "eating the elephant one bite at a time." Given the urgency of the Tier 1 systems, including those related to surgery and inpatient care, a traditional hot site contract would not meet the hospital's needs at a reasonable cost. Instead, the IS department has been enhancing and creating system redundancy for Tier 1 systems.

Crackel is now preparing to create and maintain an off-site redundant data center. She's also testing critical applications at Platform Lab, a Columbus, Ohio-based non-profit firm that provides information technology business services. This low-cost, easily accessible facility offers an ongoing resource for pre-production and disaster recovery software testing instead of relying on the industry's typical testing resources that are subscription-based and costly.

"Normally, we would have to rely on the IBM vendor to provide test time. Their contract normally covers one week per year of test time at one of their out-of-state sites. We had estimated the cost of that vendor contract to be as much as $120,000 annually," explains Crackel. "With our ability to test at Platform Lab, we can contract (with the IBM vendor) only for Tier 2 quick shipped replacement equipment. And because we've "right-sized" our program and did not put Tier 1 equipment on a vendor contract, we've realized an operational savings of more than 80%."

Crackel also notes that one week of testing availability outside of the local area is not reasonable for testing 250 servers or involving 100 information services staff members. She is "convinced that we are being smarter by using Platform Lab. We've improved on the traditional paradigm."

Children's Hospital has plans to eventually test nearly all of its applications and disaster recovery procedures at Platform Lab. "Testing is an important part of a (disaster recovery) plan. Even though we work in IS, children are at the center of what we do. They are never out of our minds. Executing regular tests gives us faith that we could continue patient care and keep children safe even in the event of a large-scale disaster," says Crackel.

And that level of commitment resonates with the grown-up in all of us.

For more information please call (614) 675-3711 or Contact Us via email.