Data Migration

The Plot 60 Way

Data migration is a key consideration for any Acturis platform implementation and is the process of selecting, preparing, extracting, and transforming data and permanently transferring data from your existing systems to the new platform. Additionally, the validation of migrated data for completeness and the decommissioning of legacy data storage are considered part of a typical data migration process.

We, keep in mind the seven “R’s” of data migration, which must have the following essential characteristics:
  • Robust & Resilient: manage all aspects of the data extraction, transformation, cleansing, validation and loading into the Acturis platform, and using processes able to manage high volumes of data & adjust to issues in the environment such as database connections, disk space and memory problems
  • Rapid: execute efficiently to enable rapid processing
  • Reporting & Reconciliation: provide progress indicators during migration and reconcile the completed process
  • Recoverable: able to restart from the point of failure when necessary.
  • Reusable: ability to reuse components of the migration in other projects, including transformation functions, error handling and data cleansing routines

Discovery

The first phase of a data migration project is to identify and explore the source systems. The most appropriate route for identification is to group data, customer names, addresses and product descriptions based on the target model.

Although the source systems may contain thousands of fields, some might be duplicates or not be applicable to the target system. In this stage, it is critical to identify which data is required and where it is, as well as what data is redundant and not required for the migration.

Conversely, if the initially identified sources do not contain all of the data required for the target model, a gap is identified. In this case, you may have to consolidate data from multiple sources to create a record with the correct set of data to fulfill the requirements of the target.

Using multiple data sources allows you to add another element of data validation and a level of confidence in your data.

At the end of this phase, you will have identified the source data that will populate the target model. You will also have identified any gaps in the data and, if possible, included extra sources to compensate. Optimally, you will have broken down the data into categories that enable you to work on manageable and possible parallel tasks.

Data assessment

The next logical phase is to assess the quality of this source data. If the new system fails due to data inconsistencies, incorrect or duplicate data, there is very limited value in migrating data to the target system. To assess the data, we recommend profiling the data.

Data profiling is the process of systematically scanning and analysing the contents of all the columns in tables of interest. Profiling identifies data defects at the table and column level. Data profiling is integral to the process of evaluating the conformity of the data and ensuring compliance to the requirements of the target system.

The profiling functions include examining the actual record value and its metadata information. Too many data migration initiatives begin without first examining the quality levels of the source data. By including data profiling early in the migration process, the risks of project overruns, delays and potentially complete failures are reduced.

Through the use of data profiling, you can:

  • Immediately identify whether the data will fit the business purpose
  • Accurately plan the integration strategy by identifying data anomalies up front
  • Successfully integrate the source data using an automated data quality process
The output of this phase of the project is a thorough understanding of the data quality in the source systems, identification of data issues and a list of defined rules to be built to correct them. You will have identified and defined your data quality rules and mappings from the sources to the target model. At this point you will also have a good idea, at a high level, of the design of the integration processes.

Migration design

In the migration design phase, the main tasks will be to define the technical architecture and design of the migration processes. In addition, you will define the testing processes and how to transition them to the production system. You will also determine whether there will be a parallel run, a zero-downtime migration, or whether you will be expecting to complete the migration and simply “decommission” the old system.

In this phase, it’s important to put your plans for the next three steps down on paper. Include timelines, technical details and any other concerns or approval requirements so that the entire project will be documented.

Migration build

Be careful about using a “just enough” development approach with your data migration, simply because the migration will be executed only once and there will be limited reuse of the code. These assumptions explain why data migrations are prone to failure.

A typical sequence to follow when developing a migration is to subset the data and test one category of data at a time (e.g., product or customer). This approach aligns with the first stage of the project which through data exploration categorises the data. In the case of larger projects, you can develop and test each category in parallel. Testing the migration solution is usually an iterative approach. Start by checking the components individually in small subsets to ensure the mappings, transformations and data cleansing routines are working. Then, increase the data volumes and eventually link all of the components together into a single migration job.

Output from this phase results in a fully tested data migration process that is scalable, reliable and can deliver the migration within the allocated time.

Execution

After comprehensive testing, the time comes to run the migration. In the majority of cases, the source systems are shut down while the migration executes. To minimise impact, this is likely to occur overnight, over a weekend or public holiday. In some cases, where the source applications are required to run 24/7 a zero-downtime migration approach may be needed. This will require additional design, development and testing effort in order provide the initial migration processes in a manner that can capture and synchronise any subsequent changes that took place.

Transition

During the execution phase, audit trails and logs will be created to ensure that all data has been correctly migrated and, when appropriate, that the correct synchronisation has been achieved. Finally, after reviewing the audit trails and logs, you will be prepared to make an informed decision to transition users to the new Acturis platform and progress your legacy system retirement plans.