It's 9:30 a.m. on a Tuesday, and production comes to a stop, the silent alarm starts going off as end users start realizing their business-critical applications have stopped working, to make things worse file shares, phones, internet and email are no longer working. With email down, the breakdown of communications begins as executives and IT leaders frantically begin communicating using insecure personal devices and non-enterprise mediums such as personal emails. With every second that goes by, revenue is being lost as co-workers begin to draw conclusions. The road to this chaotic state always differs, but they are all damaging without a contingency plan: sudden & unplanned hardware or software failures, misconfigured redundancy in network configurations, malware, and ransomware attacks.
Inevitably, when this situation occurs, there are ultimately one of two outcomes that follow: you either have an infrastructure design and disaster recovery plan that can sustain the blow, allowing business-critical applications to remain operational, or you don’t; when you don’t, you are typically looking at long periods of unplanned downtime and expenses that vary.
Unfortunately, these critical hours are usually when manufacturers discover the difference between just having backups and having a full disaster recovery plan. A successful backup doesn't guarantee a successful recovery. A restoration may take longer than expected or even fail altogether. Organizations can improve outcomes by establishing and rigorously testing the entire recovery process before an outage occurs. To help clients, prospects, and stakeholders navigate these challenges, Koltiv has provided a summary of the key details below.
Getting production up and running again involves more than restoring a server. Employees will always need access to all critical business applications, whether it's manufacturing & accounting, supply chain, sales & customer relations, manufacturing & automation, or human resources. Until all systems are working, the outage isn't over.
That's what IT disaster recovery is all about. It's the process of restoring the technology a manufacturer relies on after an unplanned or unexpected outage. Outages can be caused by any number of events, such as ransomware, a hardware failure, human error, or severe weather. No manufacturer is immune.
Before evaluating disaster recovery readiness, it's important to establish the scope of the effort. Decide which locations and systems will be included in each phase of the recovery process. Manufacturers often prioritize the systems that would immediately affect production or shipping if they became unavailable. Other systems may be important, but they don't always have the same urgency.
Next, determine who will participate in the recovery process. IT personnel, plant leadership, software vendors, and other technology partners all play different roles during an outage. Those responsibilities should be clearly understood and practiced before there's an outage.
Recovery targets should also be established for each critical system. Key metrics include Recovery Time Objective (RTO) and Recovery Point Objective (RPO). RTO is how quickly a system needs to be back up and running. For example, a manufacturer may decide the ERP system needs to be restored within two hours so production can start back up. RPO is how much data the business can afford to lose. For example, if the ERP system is backed up every hour and fails at 2:45 p.m., up to 45 minutes of data could be lost. If that isn't acceptable, backups need to occur more frequently.
The recovery targets for an ERP system will usually be much different than the recovery targets for an accounting application or another less critical system. Defining those targets ahead of time makes it easier to decide what gets restored first when an outage occurs.
Start by identifying every system that supports the operation. Then prioritize them according to business impact. If this system became unavailable, would production stop, would shipping stop, or would business continue with minimal disruption? Those answers should help determine recovery priorities.
Review the backup strategy for every critical system identified in Phase 1. Just as importantly, make sure those backups are protected. For example, if ransomware encrypts production systems, it shouldn't be able to encrypt the backups as well. Immutable or offline backups are an extra line of protection.
Walk through a few realistic outage scenarios with the key people and vendors involved in disaster recovery. Discuss how decisions will be made and how information will be communicated with employees and other stakeholders.
Run a practice test in an isolated environment to see if critical systems can be restored by following the steps in the disaster recovery plan. The goal is to verify that the plan works within the recovery targets before it is ever needed.
A server coming back online is the first step, but it doesn't necessarily mean the business is ready to operate as usual. Measure the actual recovery time against established RTOs, and verify that all systems and apps function correctly. Confirm data integrity, user authentication, and system integrations.
Record what worked and how long recovery is required. Document any problems and what improvements need to be made before the next review. Also, note who is responsible for follow-up.
After working through the phases, take a step back and evaluate the results.
Pass means recovery objectives were met and critical systems were restored to support normal business operations.
Conditional Pass means recovery was successful but issues were found that should be addressed before the next review.
Fail means one or more critical systems couldn't be restored within the RTO or didn't function as expected after recovery.
No matter the score, disaster recovery is an ongoing process. Manufacturers often review critical systems quarterly, and they are encouraged to perform a company-wide recovery exercise on an annual basis. Manufacturers usually revisit the plan after major infrastructure changes or cybersecurity incidents as well.
Finally, document the results. Recovery times, RTOs, RPOs, corrective actions, and review dates provide valuable information for leadership and may also support cyber insurance requirements.
Disaster recovery readiness is about protecting the business and restoring the systems that get production back on track when an unexpected outage occurs. It often involves a multi-phased approach and an outside perspective. If you'd like help evaluating your disaster recovery readiness or strengthening your managed IT environment, Koltiv can help. For additional information call 515.223.0078 or click here to contact us. We look forward to speaking with you soon.