What is data migration?
By definition, data migration is the practice of moving data from one system to another. This sounds simple, but data migrations are often challenging because they are complicated by the way embedded data storage systems develop over time, a wide array of data dependencies and the unique requirements that businesses have for their data.
This article discusses the different types of data migration, common reasons for migrating data, the risks and challenges of a migration and the key elements of a robust data migration strategy to ensure a successful transfer.
Why do we need to migrate data?
From time to time, information needs to be moved, either due to the inadequacies of its current environment or because a business needs to expand what it can do with the data it has collected. Some of the most common reasons that an organisation might undergo a data migration include:
Updating or improving business processes
‘Business processes’ is a broad category, and includes all the regular, structured, day-to-day activities that support the business in achieving its business goals effectively. Among these tasks there are many that are repetitive and high-frequency, which makes them excellent candidates for streamlining or automating altogether. To improve business processes, an organisation might migrate data storage from paper to digital, might move data stored in email archives to more dynamic client databases, or might move from one provider of software as a service to another to facilitate specific automated workflows.
Mergers or acquisitions
When a business is purchased and becomes part of another, there can be significant friction between teams and cultures. Without careful management, there can also be friction when it comes to technology use and data governance. Although sometimes it is tempting to permit an organisation to run off multiple systems riddled with data quality issues as it grows and expands, it typically leads to chaos in the longer term. It is often necessary to perform some kind of migration to ensure information is stored in the same places, usable and not duplicated across the business — ideally as early as practicable, to minimise technical debt and prevent exposure to data risk.
Regulatory compliance
Sometimes new regulations demand a data migration. Perhaps a type of data is reclassified, or the physical location of stored data becomes a regulatory risk, or a provider of particular services no longer meets the security needs of the business. In such cases, businesses need to undertake a data migration to ensure they retain full compliance with applicable regulations. While large changes like these usually come with a grace period for compliance, it can nevertheless impose a challenging timeline in which to execute the migration!
Replacing outdated systems
The lifecycle of most storage systems is not infinite. Businesses, especially those with a need to store data on-premises, may need to eventually undertake a data migration on this basis. However, systems can become outdated for many reasons, not just through the vagaries of time. Over time, an environment may become a less optimal fit for a business — for example, a particular software may no longer be supported by its original creator, and this could create security risks. Alternatively, it’s possible that a service provider may have grown in a direction that is no longer aligned with the needs or values of the data owner.
Moving between data centres
Occasionally, even a cloud-first business will need to rethink where its data is stored. Data centres vary in terms of geographic security, efficiency and cost-effectiveness and may provoke data sovereignty concerns. For any of these reasons, a business may need to consider a migration from one data centre to an alternative one.
Service improvements
Most businesses seek to improve services, and those which store data often use it to drive aspects of their services. Service improvements can mean migrating data in order to execute the same tasks faster, or it can mean undertaking a data migration in order to facilitate new ways of working, such as moving from batch data analytics to real time.
Facilitating growth
A data migration can also be part of a process of scaling up a business. Sometimes this is just a matter of size — environments appropriate for a small business’s database may begin to incur much greater costs as the quantity of data grows. In other cases, scaling up calls for more sophisticated use of data and records, such as the expanded tools available in a mid-tier CRM verses the spreadsheet system with which many businesses start out.
Types of data migration
For convenience, we often categorise data migration by type, even though the requirements of a particular migration are usually individual and the different types can overlap in various ways. This makes these categories less clearly delineated than they may at first appear. A storage migration, for example, may also be a cloud migration and a database migration at the same time. With that nuance in mind, then, the broad categories most commonly used to describe data migrations are as follows:
Storage migration
A storage migration is any migration that is moving data from one storage device to another. These types of migrations are of course part of upgrading a storage system — for example, moving records from paper to digital storage — but other reasons might include archiving older (but still necessary) records outside of a system where storage is at a premium, or exchanging on-premises data storage for cloud storage.
Database migration
A database refers specifically to a collection of data arranged or organised for use. Database migration means moving data from one or more source databases to a target one, and is further split into simpler homogenous migrations, in which the data is moved between databases of the same type, and more complex heterogeneous ones, in which the source and target databases differ in type. Heterogeneous data migrations typically create a more complex challenge.
Such migrations are undertaken for reasons that might include switching to a database with more optimal resource use, retiring a legacy system or merging multiple databases’ data into one destination database to create a unified view for the business.
Application migration
An application migration occurs when a software application is moved from one computing environment to another. This might be a move from one data centre to another, or from on-premises hosting to cloud hosting, for example. Some applications are optimised for specific environments, so such a migration can also (at its more complex end) include altering the application to perform better in its new environment.
Business process migration
A business process is a connected string of tasks or activities that support the core functions of a business. A business process migration is a wide category and involves migrating data and applications in any combination to better perform those processes. Reasons for doing this might range from mergers and acquisitions to enabling the use of new technology to compete better within the market.
Cloud migration
A cloud migration is any migration that involves moving to, or between, cloud hosting solutions. A company might use public cloud services or require a private environment where the cloud infrastructure is accessible only to that company. Cloud hosting can be more easily scaled than most other methods of hosting data, and using a public cloud provider averts the need for additional resourcing for many companies. Cloud migrations might include moving data between data centres, moving from on-premises to cloud, or migrating to a hybrid system that combines on-premises and cloud.
Data migration strategy
From the outset, any migration project that commences without a strategy is doomed.
A strategy is a key tool for risk mitigation and provides a roadmap to review and troubleshoot before execution to ensure a successful migration.
With this in mind, it’s best to start out by defining exactly what ‘success’ looks like.
Planning
Start with a plan:
- Define the scope of the migration. What data needs to be moved, and how which systems will it affect?
- Consider the needs of the people within the organisation who use those systems and that data. A holistic view of the current use of the data to be migrated should inform broader planning.
- Set goals for the expected outcomes of the migration, such as the state and performance of the data in its target system. Use SMART goals wherever possible — goals that are (s)pecific, (m)easurable, (a)chievable, (r)elevant and (t)ime-bound.
- Determine the timeline, budget and overall business impact of the migration process.
The current state of the data
Planning should be followed by a comprehensive assessment of the current state of the data and systems. A data audit can be used to discover the state of the data, including dependencies and more granular details like formats, types of data and the relative sensitivity of the information, as well as discovering duplicates and data quality issues. This information, once ascertained, will inform the requirements for the migration.
Avert risk of loss
Backing up all of the data touched by the migration process is a vital step, not to be skipped or overlooked. In the event that something should go wrong, it is important that the company still retains a copy of the information. This addresses the risk of losing or damaging the data, which also averts compliance risks that might flow from that loss or damage.
Define the process
With knowledge of the goals, dependencies, business impact and scope of the data, as well as its current state, it is possible to define how the migration will proceed. This step is the point at which data mapping — determining the relationships between data elements so that they can be preserved in the destination system — must be completed.
Depending on the kind of data migration the organisation must undertake, different processes may be required. For example: traditionally, a data migration involves extracting the data, transforming it as necessary, and loading it into the target system (ETL), however, for some data analysis applications, it may be more useful to extract the data, load it into the target system, and transform it after loading (ELT) as necessary. Differences in process are highly dependent on the use and format of data being migrated.
Execute
Executing the migration occurs in the scope of all of the previous steps. If all the preceding steps have been well executed, then it is typically fairly straightforward to execute the process itself.
Post migration support and review
A migration does not end at execution: it’s important to examine the new state of data and compare it with the goals and expectations laid out during planning.
This includes validating the data in the target system to make sure it’s accurate, functioning and correct. It is also worthwhile to take a moment to review the plan and process and see if the two diverged at any point, which provides useful insights that can inform any future data projects.
Risks and challenges
Security risks
The data we handle every day can be sensitive. It’s important to protect it.
Depending on the type of data involved in a migration, the security risks and obligations can differ. Sensitive information about customers or credit card data can be particularly fraught and will be covered by specific legislation. But even if that is not the type of data being moved, most migrations will include some consideration regarding the security of the information involved.
Businesses must ensure that they meet any obligations they may have relating to encryption and the security of target systems to avert unnecessary exposure to compliance risk.
Loss of data
Loss of data is a risk with any migration. Losing data can have a massive and deleterious impact on business processes and outcomes, but it is simple to prevent: ensure data is backed up securely before commencing a data migration.
Delays
In the experience of our Data Services team, this is perhaps the single most common reason for bringing experts in midway through a migration — poor planning (or the absence of a strategy at all) causes delays and blows through an organisation’s migration budget.
This risk can be mitigated by having a comprehensive, carefully-laid plan at the outset, and can also be prevented through the use of a dedicated migration team with the right skills to complete the job correctly the first time.
Final thoughts
A data migration is just as distinct as the processes and technology stack of the business that undertakes it. That means there’s truly no one-size-fits-all data migration approach that can be enacted for every project. Despite that, defining an overall strategy and crafting a step-by-step plan offers the strongest chance of success, especially in the case of more complex migrations.
Do you have questions about your organisation’s data migration? We have answers. Contact us today.
