Migrate on-premises Apache Hadoop clusters to Azure HDInsight - motivation and benefits

This article is the first in a series on best-practices for migrating on-premises Apache Hadoop eco-system deployments to Azure HDInsight. This series of articles is for people who are responsible for the design, deployment, and migration of Apache Hadoop solutions in Azure HDInsight. The roles that may benefit from these articles include cloud architects, Hadoop administrators, and DevOps engineers. Software developers, data engineers, and data scientists should also benefit from the explanation of how different types of clusters work in the cloud.

Why to migrate to Azure HDInsight

Azure HDInsight is a cloud distribution of Hadoop components. Azure HDInsight makes it easy, fast, and cost-effective to process massive amounts of data. HDInsight includes the most popular open-source frameworks such as:

• Apache Hadoop
• Apache Spark
• Apache Hive with LLAP
• Apache Kafka
• Apache HBase

Azure HDInsight advantages over on-premises Hadoop

• Low cost - Costs can be reduced by creating clusters on demand and paying only for what you use. Decoupled compute and storage provides flexibility by keeping the data volume independent of the cluster size.

• Automated cluster creation - Automated cluster creation requires minimal setup and configuration. Automation can be used for on-demand clusters.

• Managed hardware and configuration - There's no need to worry about the physical hardware or infrastructure with an HDInsight cluster. Just specify the configuration of the cluster, and Azure sets it up.

• Easily scalable - HDInsight enables you to scale workloads up or down. Azure takes care of data redistribution and workload rebalancing without interrupting data processing jobs.

• Global availability - HDInsight is available in more regions than any other big data analytics offering. Azure HDInsight is also available in Azure Government, China, and Germany, which allows you to meet your enterprise needs in key sovereign areas.

• Secure and compliant - HDInsight enables you to protect your enterprise data assets with Azure Virtual Network, encryption, and integration with Microsoft Entra ID. HDInsight also meets the most popular industry and government compliance standards.

• Simplified version management - Azure HDInsight manages the version of Hadoop eco-system components and keeps them up to date. Software updates are usually a complex process for on-premises deployments.

• Smaller clusters optimized for specific workloads with fewer dependencies between components - A typical on-premises Hadoop setup uses a single cluster that serves many purposes. With Azure HDInsight, workload-specific clusters can be created. Creating clusters for specific workloads removes the complexity of maintaining a single cluster with growing complexity.

• Productivity - You can use various tools for Hadoop and Spark in your preferred development environment.

• Extensibility with custom tools or third-party applications - HDInsight clusters can be extended with installed components and can also be integrated with the other big data solutions by using one-click deployments from the Azure Market place.

• Easy management, administration, and monitoring - Azure HDInsight integrates with Azure Monitor logs to provide a single interface with which you can monitor all your clusters.

• Integration with other Azure services - HDInsight can easily be integrated with other popular Azure services such as the following:

  • Фабрика данных Azure (ADF)
  • Azure Blob-хранилище
  • Azure Data Lake Storage 2-го поколения
  • Azure Cosmos DB (облачная база данных)
  • База данных SQL Azure
  • Azure Analysis Services

• Self-healing processes and components - HDInsight constantly checks the infrastructure and open-source components using its own monitoring infrastructure. It also automatically recovers critical failures such as unavailability of open-source components and nodes. Alerts are triggered in Ambari if any OSS component is failed.

For more information, see the article What is Azure HDInsight and the Apache Hadoop technology stack.

Migration planning process

The following steps are recommended for planning a migration of on-premises Hadoop clusters to Azure HDInsight:

1. Understand the current on-premises deployment and topologies.
2. Understand the current project scope, timelines, and team expertise.
3. Understand the Azure requirements.
4. Build out a detailed plan based on best practices.

Gathering details to prepare for a migration

This section provides template questionnaires to help gather important information about:

• The on-premises deployment
• Сведения о проекте
• Требования Azure

On-premises deployment questionnaire

Project details questionnaire

Azure requirements questionnaire

Дальнейшие действия

Прочитайте следующую статью в этом цикле:

• Architecture best practices for on-premises to Azure HDInsight Hadoop migration