Disaster Recovery Center

Persons responsible in companies for IT repeatedly consider whether data strategic for the company is secured in the event of a serious failure in the Primary Data Center (hereinafter:  PDC). Is IT ready for “blackout”? How will the company operate then? Asked in good time in advance, these questions allow you to take actions that will protect the entire organization against serious failures of the primary data center and will start the systems in the Disaster Recovery Center (hereinafter: DRC) in a controlled and planned manner at a pre-determined time. It is particularly important for multi-branch companies with an extensive organizational structure, for which every unplanned interruption in the operation of systems entails real losses. Such actions were taken by the management of Smithfield Polska in order to increase the security of its IT systems.

Business continuity plan

Creating a good business continuity plan at Smithfield Polska, covering over 50 locations throughout Poland, was certainly not an easy task. It required a structured and methodical approach to the topic, which resulted in the development and subsequent cyclical execution of test scenarios.
They provided the basis for the development of an optimal solution.

At its headquarters, Smithfield has its own extensive data center. However, the company’s IT management was aware that storing the company’s data in one place is not a fully secure solution. Unfortunately, back-up copies do not provide a sufficient guarantee of business continuity. Recovery time, problems with adapting the environment after recovery, or reduced performance can result in business interruptions that a company of the size of Smithfield simply can’t afford.

It was therefore decided to open the second independent location for DRC – Disaster Recovery Center. SNP Poland (now All for One Poland) was chosen as the project partner. It is a company that knows Smithfield systems, specializes in SAP services and has good, proven All for One Data Centers.

Systems critical for the customer

First of all, SNP consultants together with Smithfield employees determined what resources were most important for the customer, i.e. had to be replicated to the DRC in active mode. Smithfield indicated SAP systems operating under Linux control together with the IBM DB2 database as the most important ones. Replication with IBM DB2 tools in HADR (High Availability Disaster Recovery) mode ensured high availability of critical SAP systems, using transaction log replication with the ability to run a standby database in a short time.

Continuous replication

Other selected systems are continuously replicated to the DRC by VMware vReplication. The solution allows a specific group of virtual machines to replicate to the DRC in a continuous manner with a fixed maximum delay. As a result, virtual machines remain in standby mode in the DRC, allowing them to continuously add a difference with VM in the PDC.

If Smithfield loses potentially the entire PDC and decides to run the DRC, the machines will end the replication and start working in an environment with an acceptable difference with the PDC. The entire replication environment of IBM HADR and VMware vReplication is the foundation for further steps to restore the customer’s critical systems.

The third part of the environment is made up of systems collocated in the DRC, allowing the other systems to be restored from the replicated backup at a determined time in further steps.

All replicated or collocated solutions, including prepared schedules, procedures and instructions, are the basis for Smithfield and SNP teams to work together in the event of the PDC failure.

Detailed testing scenarios

In order to meet Smithfield’s requirements and expectations, SNP (now All for One Poland) engineers, in collaboration with the system users on the Smithfield side, developed detailed scenarios for testing the DRC solution. Only a tested DRC solution guarantees business continuity in the event of the PDC failure. The DRC test schedule from the beginning to the end is a real test involving cutting off Smithfield’s facilities from the headquarters and PDC, and switching fully to the DRC at SNP.

Since the procedures were developed, there have been 4 test switches, the first one in 2016 and three more in 2017. Not only IT employees on both sides but also factory staff testing the switched environment were involved in testing.

Practice makes perfect

By using the virtualization technology along with the snapshot technology on both sides, the tests were performed in real conditions, reflecting the actual situation of PDC loss.

Each time different people in the Smithfield and SNP IT teams, as well as different people in employee teams were involved in testing. All this in order to spread the knowledge and standards of operation in case of real switching. The list of tasks to be performed in the event of the PDC failure includes about 180 tasks that must be tested by various teams of employees in order to fulfil their role. As a result of testing, the procedures, schedules and instructions were refined and the time needed to restore full functionality was shortened. The above actions ensure that Smithfield systems are well protected in the event of failure.

Robert Wikieł, IT Managing Director, Smithfield Polska

Maintaining high standards
We are the largest meat company on the Polish market and for many years we have also been a leader in exporting meat to foreign markets. We invest in technology, machinery and equipment for production plants. We focus on the development that would not possible without IT systems, since it is their role to ensure business continuity.
With a quite dispersed structure (over 50 locations throughout Poland), it is a big challenge for the IT department to ensure reliable communication with the company’s systems. The situation in which we do not have access to the systems means a paralysis of a large part of the company. We have our own CPD, which we maintain to the best of our knowledge. However, experience shows that it is good to predict what is unpredictable, so we decided to launch the Disaster Recovery Center in an independent, external location.
When choosing a partner for this project, I had many offers on my desk. The choice of SNP (now All for One Poland) was based on two factors – technical capabilities and, more importantly, the basis consultants’ knowledge enabling recovery actions to be taken. We presented our expectations and developed procedures together with SNP. But procedures are one thing, and human actions are another. Therefore, the most important issue in this project is testing. Each recovery downtime was a big lesson for everyone. At that time, we saw where the problems were, what needed to be improved, streamlined and accelerated. We wanted to increase the ranges in a shorter and shorter time. In each subsequent recovery downtime, we changed our teams, employees and drew conclusions again. We did not lower the requirements, but raised the bar.
Now, the procedures developed give us a guarantee that the key IT systems for the company will function, however we do not stop there. We need to test, because only then will we be able to say that we are really prepared in the event of failure.
Robert Wikieł, IT Managing Director, Smithfield Polska

Tomasz Wawrzonek, IT Deputy Director, All for One Poland

Security of customer systems is our top priority
The infrastructure of All for One Data Centers meets high requirements related to security and performance. Monitoring systems, business continuity plans, certified work standards are the foundations for the provision of our services. The key element is the competence of people involved in ensuring business continuity and the availability of systems 24 hours a day, 7 days a week, which is why it is so important to continuously improve the competence.
The Disaster Recovery Center service is provided in the event of major failures or other unforeseeable events. If such a critical event, resulting in the unavailability of IT systems in the current operation, occurs, an authorized person from Smithfield makes a decision to switch the systems to the disaster recovery center in accordance with a pre-established procedure. After such a signal, the procedure of switching to the disaster recovery center, i.e. the business continuity plan, is activated. As a result, the disaster recovery center at SNP (now All for One Poland) will take over the role of the primary data center at the planned time. All system users will receive a message about the switching with the instructions to use “backup” connections that will enable logging into the systems.
As part of the Disaster Recovery Center service, SNP also provides Smithfield with a backup infrastructure collocation service, which enables the recovery of archival data.
Tomasz Wawrzonek, IT Deputy Director, All for One Poland