SAP migration

Big organizations, operating in many countries where they have subsidiaries, factories or representative offices, aim at unification of business processes and IT systems that support the processes. Many of them think that advantages of one corporate ERP system template, which is maintained and administrated centrally, outweigh the possible related inconveniences and limits for local subsidiaries in particular countries. That is why in an international SAP environment system rollouts are ranked as the most frequently carried out projects.

SAP landscape in Valeant

Valeant Pharmaceuticals, Int. was operating according to a similar scheme. Its factories in Mexico, Brazil and Poland were using one SAP system template, which was located on servers in a data processing center in California. System management and administration as well as coordination of changes were performed by the corporate IT department staff. In total, there are 700 people using SAP solutions in Valeant.

Within the scope of Valeant corporate solution there are the most important SAP functional areas: Finance (FI), Controlling (CO), Sales and Distribution (SD), Materials Management (MM), Warehouse Management (WM), Production Planning (PP), Quality Management (QM) and Human Resources (HR). SAP BI reporting tools are used as well.

Moreover, SAP tools for corporate governance management (SAP GRC) function as a separate installation. This rarely used in Poland functionality, relating to verification of access to key information resources and internal audits, helps to comply with the American Sarbanes-Oxley Act (SOX) for corporations registered in U.S. Securities and Exchange Commission.

GRC is used in Valeant particularly for risk management. It is worth pointing out that ICN Polfa Rzeszów, as a part of Valeant corporation, will be still subject to rules and requirements established by SOX. SAP GRC will provide support for ICN in this matter. Another area, in which SAP GRC tools are used in the company, is a system of authorizations and access control to key company information resources.

From California to Rzeszów

In 1997 Rzeszowskie Zakłady Farmaceutyczne “Polfa” became a part of ICN Pharmaceuticals Inc. (now Valeant Pharmaceuticals International). Since then the company has had good financial results and it has been effectively competing with the biggest pharmaceutical producers in Poland and abroad. At the beginning of 2008 a SAP rollout project was completed in ICN Polfa Rzeszów and the company was covered by the corporate SAP Valeant system template. The system was administrated centrally and the IT team and key users from Rzeszów were responsible for preparing functional changes and testing.

It had been true up until mid-2009. After nearly 2 years of functioning of a global SAP system in Valeant, a decision on separating a system for the Polish subsidiary and making it independent of the corporate template was made.

Why separately?

From the system administration perspective one global solution may be beneficial in terms of e.g. solution coherence, lower maintenance and development costs and lower costs of access to competence.

It is not that obvious when we take business processes into account. The Valeant subsidiary in Poland constitutes a considerable part of business, corporation-wide. ICN Polfa Rzeszów operates business that is independent of subsidiaries in Latin America. All that makes ICN Polfa’s business expectations toward SAP system differ from the expectations in other subsidiaries.

In pharmaceutical industry the procedures relating to system development and its adjustment to current needs are centralized and quite formalized, which stem from the specificity of the business, and it meant that our hands were rather tied.

Any small change in SAP had to be preceded with all subsidiaries’ acceptance and detailed documentation, which lengthened the process. This was a direct cause of separating the system for the purposes of ICN Polfa Rzeszów.

Other reasons for SAP system separation and transferring a part of it to Rzeszów were mostly operational ones and they included the following:

• different time zones in which the particular subsidiaries and the California Data Center operate, which caused work organisation difficulties and technical problems relating to change management;
• supervising changes so that a valid system status is maintained required countless hours of arrangements and testing to see if the solution that works properly at one manufacturing plant will not cause any problems in the other;
• to be able to support business users efficiently, SAP system needs to develop, and that was hindered since every change, even the smallest one, had to be accepted by all subsidiaries;
• assigning authorizations often meant waiting a few hours instead of a few minutes;
• similarly to changes management, problem reporting was also difficult due to time difference between California and Poland. Hence the time of handling messages was longer (in fact, different working hours made it impossible to communicate more than once a day);
• system efficiency was not considered a big problem, however, we expected that after the migration process the users would notice an improvement

SAP in 1:1 scale

Migration projects of SAP systems, like the above quoted one, are complicated undertakings where there have to be many aspects taken into account, such as the ones relating to system functionality, its technical layer and basic data. Paradoxically, the results of this undertaking are imperceptible for the users – configuration and functionalities remain the same.

In ICN Polfa Rzeszów the goal of the project was to create new systems that would be a 1:1 mapping of the configuration of the source system in the U.S. and to execute SAP validation including changes from the period of the last two years.

The project covered SAP ERP systems migration in ECC 5.0 version, which is still used in the corporation, as well as an installation of SAP BI and SAP GRC systems in new, upgraded software versions, keeping all configuration settings from the template. In addition, SAP Solution Manager was implemented.

Looking at the migration process from a technical perspective, an export of data from SAP ERP production system was done, on the basis of which a development system, a testing system, a training system and a production system were created. The entire migration process was executed according to the Heterogeneous System Copy scenario for SAP. The transfer was carried out by means of the above mentioned method due to the fact that a change in the operating system was made during the data migration. SAP ERP systems in the Californian data center work using HP-UX operating system, while SAP systems in the ICN Polfa Rzeszów data center were installed based on Microsoft Windows operating system.

The migration process was different from a regular approach to SAP system migration, as on the basis of data exported from production system, a development system was installed, keeping the following: a base of object versions in SAP repository, templates of authorization, roles, authorization profiles, users definitions.

Another interesting task during SAP ERP systems migration was the conversion of logical client name in the production system. The process involves risk of business data inconsistency therefore it was carried out with close cooperation between BCC Consultants and ICN Polfa Rzeszów’s SAP Module Consultants. Moreover, a series of tests was carried out to eliminate any errors relating to the conversion of logical client name.

Due to demanding security requirements, specified for SAP systems in ICN Polfa, an analysis of SAP system security measures was made, as a result of which the system configuration was made in order to eliminate security gaps.

It is worth pointing out that a three-system SAP environment is very common, whereas a separate training environment is rarely created. In ICN Polfa it was decided to create the system as an additional security barrier, so that no invalidated or unauthorized changes could get into the validated systems (i.e. testing and production systems). SAP BI system has development, testing and production versions.

Each version is further divided into SAP NetWeaver Portal and ABAP applications. Finally, SAP GRC that is maintained in a development and testing version and in a production version was installed during the project and integration with SAP ERP systems was carried out as well as basic configuration of GRC environment.

Additionally, SAP Solution Manager environment was installed and configured and it currently serves for collecting information on systems and for Early Watch Reports preparing, as well as for proactive and reactive analysis of SAP systems’ work in ICN Polfa Rzeszów.

ICN Polfa IT staff members were responsible for system migration in the module part (business) of the project; they were also conducting tests in this matter. BCC was a partner on this project and BCC Consultants carried out tasks relating to the technical (Basis) part of the project.

BCC was also responsible for data export from the system in the U.S. At first, these tasks were to be conducted by the SAP team in California but in the end it was decided that, considering the experience in execution of such projects, the task would be conducted by BCC, remotely.

Since February 2010, a landscape of SAP solutions that is separate and independent of the corporate template has been in operation in ICN Polfa Rzeszów.

Since going live, the system in Rzeszów has been working independently and it will be developed in accordance with the company’s business needs in Europe.

If it is not documented, it does not exist

In a pharmaceutical company validation and documenting activities are a part of the routine. However, never before has the IT department performed them to such a large extent as when it happened with SAP system migration.

The intense pressure on verification of project works resulted from the pharmaceutical industry specificity and legal requirements accompanying it. Good Manufacturing Practice standards (GMP) define requirements applicable to production and distribution of medicines precisely. In pharmaceutical industry these standards must be adhered to and they are regulated by a number of legal acts. QA systems require that IT solutions meet the GMP conditions and that each step of the company’s process can be proved and documented.

Data transfer process had to guarantee 100% certainty that everything was running properly. It was not the end of challenges, though. One of the basic GMP rules says: “If it’s not written down then it didn’t happen!”. Consequently, there were thousands of documents created during the project.

Functional teams put in considerable effort in preparing user requirements and functional documentation. On the grounds of that, over a thousand scripts of unit tests (OQ) and integration tests (PQ) were created. Other types of documents were created in smaller numbers, still, they were labour-intensive and required lengthy discussions and arrangements.

Data migration and starting a new SAP ERP system was preceded by tests which confirmed that all technical and business processes would be running smoothly.

During nearly a 6 months long project, the corporate system, which is used also in the U.S. and Latin America, was alive and being developed. This significantly impeded running tests and required implementation of change management process in order to be able to map the 1:1 installation. Only in the last stage, a month before the end of the project, the change process was frozen to facilitate final tests.

The testing itself was running without disruptions and was conducted in two stages, so that there was only one thing left to do during the third data import into the production system – the verification of the very migration process (SMP).

All migration activities were synchronized with subsequent validation stages. The expertise of the team members and their experience of over 2 years in the everyday use of SAP system was invaluable. On account of that the challenges we faced during the project (audits and an inventory process) did not delay the planned date of project completion.

During the project there were technical audits on migration works. Apart from BCC and ICN staff, independent experts were engaged in validation activities and technical audits.

For BCC Consultants the necessity of validation of processes was also connected with additional tasks. Long before the project started, they were being trained in internal procedures on creating documents according to the ICN standard.

Easier communication and shorter approval path

When the systems were started on February 15, 2010, the project team could see for themselves that the work was well done. The users who logged on to SAP after start up did not notice any change apart from the fact that the system responded to their commands 3 or 4 times faster.

Smooth migration and the lack of any migration-related errors was a credit to cooperation of all people directly engaged in the project and also those who supported it. The experience gained by the team will be definitely put in use in change management process and other SAP-related tasks.

ICN Polfa Rzeszów has now a fully functional and independent SAP system. Although its current functional range and configuration constitute a mapping of source installation, SAP ERP as well as SAP BI and SAP GRC have been able to be developed independently of the template since the migration moment, according to business expectations and technical needs in ICN Polfa Rzeszów.

Module areas of the system are maintained by a highly competent SAP team in ICN Polfa Rzeszów. Its IT department coordinates the development works and consults the corporate IT only on the overall strategy regarding development and SAP maintenance.
ICN Polfa Rzeszów decided to cooperate with BCC Outsourcing Center when it comes to system administration.

Since the end of February 2010, BBC Consultants have been providing remote SAP ERP, SAP BI and SAP GRC administration. BCC Outsourcing Center is the primary support line for the IT Department in ICN Polfa Rzeszów. Within the scope of the agreement, BCC Consultants do reactive work connected with service requests sent by ICN IT department and proactive work aiming at maintaining stability and security of SAP systems work. The proactive activities include SAP system monitoring and database monitoring, SAP SM Early Watch Reports analysis and system configuration for ensuring its higher work efficiency.

Among BCC tasks there is also taking efficient corrective actions for incidents logged in database system log and for SAP system as well as installing patches on SAP system and database.

The system servers are located in Rzeszów, within the same local network as the end users’ network. System development and administration is performed in one time zone, which has made communication a lot easier and has shortened the change approval path (although the strict change management system was maintained).

Communication on technical administration of the system is much easier now. IT staff have constant access to Basis competence – the same working hours in ICN Polfa Rzeszów and BCC Outsourcing Center turned out to be very convenient. Service requests are handled in compliance with the strict SLA conditions laid out in the outsourcing agreement.