Disaster recovery plan

A disaster recovery plan (DRP) is a documented process or set of procedures to recover and protect a business IT infrastructure in the event of a disaster.^[1] Such a plan, ordinarily documented in written form, specifies procedures an organization is to follow in the event of a disaster. It is "a comprehensive statement of consistent actions to be taken before, during and after a disaster."^[2] The disaster could be natural, environmental or man-made. Man-made disasters could be intentional (for example, an act of a terrorist) or unintentional (that is, accidental, such as the breakage of a man-made dam).

Given organizations' increasing dependency on information technology to run their operations, a disaster recovery plan, sometimes erroneously called a Continuity of Operations Plan (COOP), is increasingly associated with the recovery of information technology data, assets, and facilities.

Objectives

Organizations cannot always avoid disasters, but with careful planning the effects of a disaster can be minimized. The objective of a disaster recovery plan is to minimize downtime and data loss.^[3] The primary objective is to protect the organization in the event that all or part of its operations and/or computer services are rendered unusable. The plan minimizes the disruption of operations and ensures that some level of organizational stability and an orderly recovery after a disaster will prevail.^[2] Minimizing downtime and data loss is measured in terms of two concepts: the recovery time objective (RTO) and the recovery point objective (RPO).

The recovery time objective is the time within which a business process must be restored, after a major incident (MI) has occurred, in order to avoid unacceptable consequences associated with a break in business continuity. The recovery point objective (RPO) is the age of files that must be recovered from backup storage for normal operations to resume if a computer, system, or network goes down as a result of a MI. The RPO is expressed backwards in time (that is, into the past) starting from the instant at which the MI occurs, and can be specified in seconds, minutes, hours, or days.^[4] The recovery point objective (RPO) is thus the maximum acceptable amount of data loss measured in time. It is the age of the files or data in backup storage required to resume normal operations after the MI.^[5]

Relationship to the Business Continuity Plan

According to the SANS institute, the Business Continuity Plan (BCP) is a comprehensive organizational plan that includes the disaster recovery plan. The Institute further states that a Business Continuity Plan (BCP) consists of the five component plans:^[6]

• Business Resumption Plan
• Occupant Emergency Plan
• Continuity of Operations Plan
• Incident Management Plan
• Disaster Recovery Plan

The Institute states that the first three plans (Business Resumption, Occupant Emergency, and Continuity of Operations Plans) do not deal with the IT infrastructure. They further state that the Incident Management Plan (IMP) does deal with the IT infrastructure, but since it establishes structure and procedures to address cyber attacks against an organization’s IT systems, it generally does not represent an agent for activating the Disaster Recovery Plan, leaving The Disaster Recovery Plan as the only BCP component of interest to IT.^[6]

Disaster Recovery Institute International states that disaster recovery is the area of business continuity that deals with technology recovery as opposed to the recovery of business operations.^[7]

Benefits

Like every insurance plan, there are benefits that can be obtained from the drafting of a disaster recovery plan. Some of these benefits are:^[2]

• Providing a sense of security
• Minimizing risk of delays
• Guaranteeing the reliability of standby systems
• Providing a standard for testing the plan
• Minimizing decision-making during a disaster
• Reducing potential legal liabilities
• Lowering unnecessarily stressful work environment

Types of plans

There is no one right type of disaster recovery plan,^[8] nor is there a one-size-fits-all disaster recovery plan.^[1][8] However, there are three basic strategies that feature in all disaster recovery plans: (1) preventive measures, (2) detective measures, and (3) corrective measures.^[9] Preventive measures will try to prevent a disaster from occurring. These measures seek to identify and reduce risks. They are designed to mitigate or prevent an event from happening. These measures may include keeping data backed up and off site, using surge protectors, installing generators and conducting routine inspections. Detective measures are taken to discover the presence of any unwanted events within the IT infrastructure. Their aim is to uncover new potential threats. They may detect or uncover unwanted events. These measures include installing fire alarms, using up-to-date antivirus software, holding employee training sessions, and installing server and network monitoring software. Corrective measures are aimed to restore a system after a disaster or otherwise unwanted event takes place. These measures focus on fixing or restoring the systems after a disaster. Corrective measures may include keeping critical documents in the Disaster Recovery Plan or securing proper insurance policies, after a "lessons learned" brainstorming session.^[1][10]

A disaster recovery plan must answer at least three basic questions: (1) what is its objective and purpose, (2) who will be the people or teams who will be responsible in case any disruptions happen, and (3) what will these people do (the procedures to be followed) when the disaster strikes.^[11]

Types of disasters

The tsunami that affected Japan in 2011, a type of natural disaster

September 11, 2001, in New York City, a type of man-made disaster: it caused pollution, loss of lives, property damage, and considerable data loss

Disasters can be natural or man-made. Man-made disasters could be intentional (for example, sabotage or an act of terrorism) or unintentional (that is, accidental, such as the breakage of a man-made dam). Disasters may encompass more than weather. They may involve Internet threats or take on other man-made manifestations such as theft.^[1]

Natural disaster

A natural disaster is a major adverse event resulting from the earth's natural hazards. Examples of natural disasters are floods, tsunamis, tornadoes, hurricanes/cyclones, volcanic eruptions, earthquakes, heat waves, and landslides. Other types of disasters include the more cosmic scenario of an asteroid hitting the Earth.

Man-made disasters

Man-made disasters are the consequence of technological or human hazards. Examples include stampedes, urban fires, industrial accidents, oil spills, nuclear explosions/nuclear radiation and acts of war. Other types of man-made disasters include the more cosmic scenarios of catastrophic global warming, nuclear war, and bioterrorism.

The following table categorizes some disasters and notes first response initiatives. Note that whereas the sources of a disaster may be natural (for example, heavy rains) or man-made (for example, a broken dam), the results may be similar (flooding).^[12]

In the realm of information technology per se, disasters may also be the result of a computer security exploit. Some of these are: computer viruses, cyberattacks, denial-of-service attacks, hacking, and malware exploits. These are ordinarily attended to by information security experts.

Planning methodology

According to Geoffrey H. Wold of the Disaster Recovery Journal, the entire process involved in developing a Disaster Recovery Plan consists of 10 steps:^[2]

Obtaining top management commitment

For a disaster recovery plan to be successful, the central responsibility for the plan must reside on top management. Management is responsible for coordinating the disaster recovery plan and ensuring its effectiveness within the organization. It is also responsible for allocating adequate time and resources required in the development of an effective plan. Resources that management must allocate include both financial considerations and the effort of all personnel involved.

Establishing a planning committee

A planning committee is appointed to oversee the development and implementation of the plan. The planning committee includes representatives from all functional areas of the organization. Key committee members customarily include the operations manager and the data processing manager. The committee also defines the scope of the plan.

Performing a risk assessment

The planning committee prepares a risk analysis and a business impact analysis (BIA) that includes a range of possible disasters, including natural, technical and human threats. Each functional area of the organization is analyzed to determine the potential consequence and impact associated with several disaster scenarios. The risk assessment process also evaluates the safety of critical documents and vital records. Traditionally, fire has posed the greatest threat to an organization. Intentional human destruction, however, should also be considered. A thorough plan provides for the “worst case” situation: destruction of the main building. It is important to assess the impacts and consequences resulting from loss of information and services. The planning committee also analyzes the costs related to minimizing the potential exposures.

Establishing priorities for processing and operations

At this point, the critical needs of each department within the organization are evaluated in order to prioritize them. Establishing priorities is important because no organization possesses infinite resources and criteria must be set as to where to allocate resources first. Some of the areas often reviewed during the prioritization process are functional operations, key personnel and their functions, information flow, processing systems used, services provided, existing documentation, historical records, and the department's policies and procedures.

Processing and operations are analyzed to determine the maximum amount of time that the department and organization can operate without each critical system. This will later get mapped into the Recovery Time Objective. A critical system is defined as that which is part of a system or procedure necessary to continue operations should a department, computer center, main facility or a combination of these be destroyed or become inaccessible. A method used to determine the critical needs of a department is to document all the functions performed by each department. Once the primary functions have been identified, the operations and processes are then ranked in order of priority: essential, important and non-essential.

Determining recovery strategies

During this phase, the most practical alternatives for processing in case of a disaster are researched and evaluated. All aspects of the organization are considered, including physical facilities, computer hardware and software, communications links, data files and databases, customer services provided, user operations, the overall management information systems (MIS) structure, end-user systems, and any other processing operations.

Alternatives, dependent upon the evaluation of the computer function, may include: hot sites, warm sites, cold sites, reciprocal agreements, the provision of more than one data center, the installation and deployment of multiple computer system, duplication of service center, consortium arrangements, lease of equipment, and any combinations of the above.

Written agreements for the specific recovery alternatives selected are prepared, specifying contract duration, termination conditions, system testing, cost, any special security procedures, procedure for the notification of system changes, hours of operation, the specific hardware and other equipment required for processing, personnel requirements, definition of the circumstances constituting an emergency, process to negotiate service extensions, guarantee of compatibility, availability, non-mainframe resource requirements, priorities, and other contractual issues.

Collecting data

In this phase, data collection takes place. Among the recommended data gathering materials and documentation often included are various lists (employee backup position listing, critical telephone numbers list, master call list, master vendor list, notification checklist), inventories (communications equipment, documentation, office equipment, forms, insurance policies, workgroup and data center computer hardware, microcomputer hardware and software, office supply, off-site storage location equipment, telephones, etc.), distribution register, software and data files backup/retention schedules, temporary location specifications, any other such other lists, materials, inventories and documentation. Pre-formatted forms are often used to facilitate the data gathering process.

Organizing and documenting a written plan

Next, an outline of the plan’s contents is prepared to guide the development of the detailed procedures. Top management reviews and approves the proposed plan. The outline can ultimately be used for the table of contents after final revision. Other four benefits of this approach are that (1) it helps to organize the detailed procedures, (2) identifies all major steps before the actual writing process begins, (3) identifies redundant procedures that only need to be written once, and (4) provides a road map for developing the procedures.

It is often considered best practice to develop a standard format for the disaster recovery plan so as to facilitate the writing of detailed procedures and the documentation of other information to be included in the plan later. This helps ensure that the disaster plan follows a consistent format and allows for its ongoing future maintenance. Standardization is also important if more than one person is involved in writing the procedures.

It is during this phase that the actual written plan is developed in its entirety, including all detailed procedures to be used before, during, and after a disaster. The procedures include methods for maintaining and updating the plan to reflect any significant internal, external or systems changes. The procedures allow for a regular review of the plan by key personnel within the organization. The disaster recovery plan is structured using a team approach. Specific responsibilities are assigned to the appropriate team for each functional area of the organization. Teams responsible for administrative functions, facilities, logistics, user support, computer backup, restoration and other important areas in the organization are identified.

The structure of the contingency organization may not be the same as the existing organization chart. The contingency organization is usually structured with teams responsible for major functional areas such as administrative functions, facilities, logistics, user support, computer backup, restoration, and any other important area.

The management team is especially important because it coordinates the recovery process. The team assesses the disaster, activates the recovery plan, and contacts team managers. The management team also oversees, documents and monitors the recovery process. It is helpful when management team members are the final decision-makers in setting priorities, policies and procedures. Each team has specific responsibilities that are completed to ensure successful execution of the plan. The teams have an assigned manager and an alternate in case the team manager is not available. Other team members may also have specific assignments where possible.

Developing testing criteria and procedures

Best practices dictate that DR plans be thoroughly tested and evaluated on a regular basis (at least annually). Thorough DR plans include documentation with the procedures for testing the plan. The tests will provide the organization with the assurance that all necessary steps are included in the plan. Other reasons for testing include:

• Determining the feasibility and compatibility of backup facilities and procedures.
• Identifying areas in the plan that need modification.
• Providing training to the team managers and team members.
• Demonstrating the ability of the organization to recover.
• Providing motivation for maintaining and updating the disaster recovery plan.

Testing the plan

After testing procedures have been completed, an initial "dry run" of the plan is performed by conducting a structured walk-through test. The test will provide additional information regarding any further steps that may need to be included, changes in procedures that are not effective, and other appropriate adjustments. These may not become evident unless an actual dry-run test is performed. The plan is subsequently updated to correct any problems identified during the test. Initially, testing of the plan is done in sections and after normal business hours to minimize disruptions to the overall operations of the organization. As the plan is further polished, future tests occur during normal business hours.

Types of tests include: checklist tests, simulation tests, parallel tests, and full interruption tests.

Obtaining plan approval

Once the disaster recovery plan has been written and tested, the plan is then submitted to management for approval. It is top management’s ultimate responsibility that the organization has a documented and tested plan. Management is responsible for (1) establishing the policies, procedures and responsibilities for comprehensive contingency planning, and (2) reviewing and approving the contingency plan annually, documenting such reviews in writing.

Organizations that receive information processing from service bureaus will, in addition, also need to (1) evaluate the adequacy of contingency plans for its service bureau, and (2)ensure that its contingency plan is compatible with its service bureau’s plan.

Caveats/controversies

Due to its high cost, disaster recovery plans are not without critics. Cormac Foster has identified five "common mistakes" organizations often make related to disaster recovery planning:^[19]

Lack of buy-in

One factor is the perception by executive management that DR planning is "just another fake earthquake drill" or CEOs that fail to make DR planning and preparation a priority, are often significant contributors to the failure of a DR plan.

Incomplete RTOs and RPOs

Another critical point is failure to include each and every important business process or a block of data. "Every item in your DR plan requires a Recovery Time Objective (RTO) defining maximum process downtime or a Recovery Point Objective (RPO) noting an acceptable restore point. Anything less creates ripples that can extend the disaster's impact." As an example, "payroll, accounting and the weekly customer newsletter may not be mission-critical in the first 24 hours, but left alone for several days, they can become more important than any of your initial problems."

Systems myopia

A third point of failure involves focusing only on DR without considering the larger business continuity needs: "Data and systems restoration after a disaster are essential, but every business process in your organization will need IT support, and that support requires planning and resources." As an example, corporate office space lost to a disaster can result in an instant pool of teleworkers which, in turn, can overload a company's VPN overnight, overwork the IT support staff at the blink of an eye and cause serious bottlenecks and monopolies with the dial-in PBX system.

Lax security

When there is a disaster, an organization's data and business processes become vulnerable. As such, security can be more important than the raw speed involved in a disaster recovery plan's RTO. The most critical consideration then becomes securing the new data pipelines: from new VPNs to the connection from offsite backup services. Another security concern includes documenting every step of the recovery process—something that is especially important in highly regulated industries, government agencies, or in disasters requiring post-mortem forensics. Locking down or remotely wiping lost handheld devices is also an area that may require addressing.

Outdated plans

Another important aspect that is often overlooked involves the frequency with which DR Plans are updated. Yearly updates are recommended but some industries or organizations require more frequent updates because business processes evolve or because of quicker data growth. To stay relevant, disaster recovery plans should be an integral part of all business analysis processes, and should be revisited at every major corporate acquisition, at every new product launch and at every new system development milestone.

See also

• Disaster recovery
• Business continuity planning
• Federal Emergency Management Agency
• Backup rotation scheme
• Seven tiers of disaster recovery

References