Copy data from the HDFS server using Azure Data Factory or Synapse Analytics

APPLIES TO: Azure Data Factory Azure Synapse Analytics

This article outlines how to copy data from the Hadoop Distributed File System (HDFS) server. To learn more, read the introductory articles for Azure Data Factory and Synapse Analytics.

Supported capabilities

This HDFS connector is supported for the following capabilities:

① Azure integration runtime ② Self-hosted integration runtime

Specifically, the HDFS connector supports:

• Copying files by using Windows (Kerberos) or Anonymous authentication.
• Copying files by using the webhdfs protocol or built-in DistCp support.
• Copying files as is or by parsing or generating files with the supported file formats and compression codecs.

Prerequisites

If your data store is located inside an on-premises network, an Azure virtual network, or Amazon Virtual Private Cloud, you need to configure a self-hosted integration runtime to connect to it.

If your data store is a managed cloud data service, you can use the Azure Integration Runtime. If the access is restricted to IPs that are approved in the firewall rules, you can add Azure Integration Runtime IPs to the allow list.

You can also use the managed virtual network integration runtime feature in Azure Data Factory to access the on-premises network without installing and configuring a self-hosted integration runtime.

For more information about the network security mechanisms and options supported by Data Factory, see Data access strategies.

Get started

To perform the Copy activity with a pipeline, you can use one of the following tools or SDKs:

• The Copy Data tool
• The Azure portal
• The .NET SDK
• The Python SDK
• Azure PowerShell
• The REST API
• The Azure Resource Manager template

Create a linked service to HDFS using UI

Use the following steps to create a linked service to HDFS in the Azure portal UI.

1. Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services, then click New:

   • Azure Data Factory
   • Azure Synapse

   

2. Search for HDFS and select the HDFS connector.

   

3. Configure the service details, test the connection, and create the new linked service.

   

Connector configuration details

The following sections provide details about properties that are used to define Data Factory entities specific to HDFS.

Linked service properties

The following properties are supported for the HDFS linked service:

Example: using Anonymous authentication

{
    "name": "HDFSLinkedService",
    "properties": {
        "type": "Hdfs",
        "typeProperties": {
            "url" : "http://<machine>:50070/webhdfs/v1/",
            "authenticationType": "Anonymous",
            "userName": "hadoop"
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

Example: using Windows authentication

{
    "name": "HDFSLinkedService",
    "properties": {
        "type": "Hdfs",
        "typeProperties": {
            "url" : "http://<machine>:50070/webhdfs/v1/",
            "authenticationType": "Windows",
            "userName": "<username>@<domain>.com (for Kerberos auth)",
            "password": {
                "type": "SecureString",
                "value": "<password>"
            }
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

Dataset properties

For a full list of sections and properties that are available for defining datasets, see Datasets.

Azure Data Factory supports the following file formats. Refer to each article for format-based settings.

• Avro format
• Binary format
• Delimited text format
• Excel format
• JSON format
• ORC format
• Parquet format
• XML format

The following properties are supported for HDFS under location settings in the format-based dataset:

Example:

{
    "name": "DelimitedTextDataset",
    "properties": {
        "type": "DelimitedText",
        "linkedServiceName": {
            "referenceName": "<HDFS linked service name>",
            "type": "LinkedServiceReference"
        },
        "schema": [ < physical schema, optional, auto retrieved during authoring > ],
        "typeProperties": {
            "location": {
                "type": "HdfsLocation",
                "folderPath": "root/folder/subfolder"
            },
            "columnDelimiter": ",",
            "quoteChar": "\"",
            "firstRowAsHeader": true,
            "compressionCodec": "gzip"
        }
    }
}

Copy activity properties

For a full list of sections and properties that are available for defining activities, see Pipelines and activities. This section provides a list of properties that are supported by the HDFS source.

HDFS as source

Azure Data Factory supports the following file formats. Refer to each article for format-based settings.

• Avro format
• Binary format
• Delimited text format
• Excel format
• JSON format
• ORC format
• Parquet format
• XML format

The following properties are supported for HDFS under storeSettings settings in the format-based Copy source:

Example:

"activities":[
    {
        "name": "CopyFromHDFS",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<Delimited text input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "DelimitedTextSource",
                "formatSettings":{
                    "type": "DelimitedTextReadSettings",
                    "skipLineCount": 10
                },
                "storeSettings":{
                    "type": "HdfsReadSettings",
                    "recursive": true,
                    "distcpSettings": {
                        "resourceManagerEndpoint": "resourcemanagerendpoint:8088",
                        "tempScriptPath": "/usr/hadoop/tempscript",
                        "distcpOptions": "-m 100"
                    }
                }
            },
            "sink": {
                "type": "<sink type>"
            }
        }
    }
]

Folder and file filter examples

This section describes the resulting behavior if you use a wildcard filter with the folder path and file name.

File list examples

This section describes the behavior that results from using a file list path in the Copy activity source. It assumes that you have the following source folder structure and want to copy the files that are in bold type:

Use DistCp to copy data from HDFS

DistCp is a Hadoop native command-line tool for doing a distributed copy in a Hadoop cluster. When you run a command in DistCp, it first lists all the files to be copied and then creates several Map jobs in the Hadoop cluster. Each Map job does a binary copy from the source to the sink.

The Copy activity supports using DistCp to copy files as is into Azure Blob storage (including staged copy) or an Azure data lake store. In this case, DistCp can take advantage of your cluster's power instead of running on the self-hosted integration runtime. Using DistCp provides better copy throughput, especially if your cluster is very powerful. Based on the configuration, the Copy activity automatically constructs a DistCp command, submits it to your Hadoop cluster, and monitors the copy status.

Prerequisites

To use DistCp to copy files as is from HDFS to Azure Blob storage (including staged copy) or the Azure data lake store, make sure that your Hadoop cluster meets the following requirements:

• The MapReduce and YARN services are enabled.

• YARN version is 2.5 or later.

• The HDFS server is integrated with your target data store: Azure Blob storage or Azure Data Lake Store (ADLS Gen1):

  • Azure Blob FileSystem is natively supported since Hadoop 2.7. You need only to specify the JAR path in the Hadoop environment configuration.
  • Azure Data Lake Store FileSystem is packaged starting from Hadoop 3.0.0-alpha1. If your Hadoop cluster version is earlier than that version, you need to manually import Azure Data Lake Store-related JAR packages (azure-datalake-store.jar) into the cluster from here, and specify the JAR file path in the Hadoop environment configuration.

• Prepare a temp folder in HDFS. This temp folder is used to store a DistCp shell script, so it will occupy KB-level space.

• Make sure that the user account that's provided in the HDFS linked service has permission to:

  • Submit an application in YARN.
  • Create a subfolder and read/write files under the temp folder.

Configurations

For DistCp-related configurations and examples, go to the HDFS as source section.

Use Kerberos authentication for the HDFS connector

There are two options for setting up the on-premises environment to use Kerberos authentication for the HDFS connector. You can choose the one that better fits your situation.

• Option 1: Join a self-hosted integration runtime machine in the Kerberos realm
• Option 2: Enable mutual trust between the Windows domain and the Kerberos realm

For either option, make sure you turn on webhdfs for Hadoop cluster:

1. Create the HTTP principal and keytab for webhdfs.

   Important

   The HTTP Kerberos principal must start with "HTTP/" according to Kerberos HTTP SPNEGO specification. Learn more from here.

   Kadmin> addprinc -randkey HTTP/<namenode hostname>@<REALM.COM>
   Kadmin> ktadd -k /etc/security/keytab/spnego.service.keytab HTTP/<namenode hostname>@<REALM.COM>
   

2. HDFS configuration options: add the following three properties in hdfs-site.xml.

   <property>
       <name>dfs.webhdfs.enabled</name>
       <value>true</value>
   </property>
   <property>
       <name>dfs.web.authentication.kerberos.principal</name>
       <value>HTTP/_HOST@<REALM.COM></value>
   </property>
   <property>
       <name>dfs.web.authentication.kerberos.keytab</name>
       <value>/etc/security/keytab/spnego.service.keytab</value>
   </property>
   

Option 1: Join a self-hosted integration runtime machine in the Kerberos realm

Requirements

• The self-hosted integration runtime machine needs to join the Kerberos realm and can’t join any Windows domain.

How to configure

On the KDC server:

Create a principal, and specify the password.

Kadmin> addprinc <username>@<REALM.COM>

On the self-hosted integration runtime machine:

1. Run the Ksetup utility to configure the Kerberos Key Distribution Center (KDC) server and realm.

   The machine must be configured as a member of a workgroup, because a Kerberos realm is different from a Windows domain. You can achieve this configuration by setting the Kerberos realm and adding a KDC server by running the following commands. Replace REALM.COM with your own realm name.

   C:> Ksetup /setdomain REALM.COM
   C:> Ksetup /addkdc REALM.COM <your_kdc_server_address>
   

   After you run these commands, restart the machine.

2. Verify the configuration with the Ksetup command. The output should be like:

   C:> Ksetup
   default realm = REALM.COM (external)
   REALM.com:
       kdc = <your_kdc_server_address>
   

In your data factory or Synapse workspace:

• Configure the HDFS connector by using Windows authentication together with your Kerberos principal name and password to connect to the HDFS data source. For configuration details, check the HDFS linked service properties section.

Option 2: Enable mutual trust between the Windows domain and the Kerberos realm

Requirements

• The self-hosted integration runtime machine must join a Windows domain.
• You need permission to update the domain controller's settings.

How to configure

On the KDC server:

1. Edit the KDC configuration in the krb5.conf file to let KDC trust the Windows domain by referring to the following configuration template. By default, the configuration is located at /etc/krb5.conf.

   [logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log
   
   [libdefaults]
    default_realm = REALM.COM
    dns_lookup_realm = false
    dns_lookup_kdc = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
   
   [realms]
    REALM.COM = {
     kdc = node.REALM.COM
     admin_server = node.REALM.COM
    }
   AD.COM = {
    kdc = windc.ad.com
    admin_server = windc.ad.com
   }
   
   [domain_realm]
    .REALM.COM = REALM.COM
    REALM.COM = REALM.COM
    .ad.com = AD.COM
    ad.com = AD.COM
   
   [capaths]
    AD.COM = {
     REALM.COM = .
    }
   

   After you configure the file, restart the KDC service.

2. Prepare a principal named krbtgt/[email protected] in the KDC server with the following command:

   Kadmin> addprinc krbtgt/[email protected]
   

3. In the hadoop.security.auth_to_local HDFS service configuration file, add RULE:[1:$1@$0](.*\@AD.COM)s/\@.*//.

On the domain controller:

1. Run the following Ksetup commands to add a realm entry:

   C:> Ksetup /addkdc REALM.COM <your_kdc_server_address>
   C:> ksetup /addhosttorealmmap HDFS-service-FQDN REALM.COM
   

2. Establish trust from the Windows domain to the Kerberos realm. [password] is the password for the principal krbtgt/[email protected].

   C:> netdom trust REALM.COM /Domain: AD.COM /add /realm /password:[password]
   

3. Select the encryption algorithm that's used in Kerberos.

   a. Select Server Manager > Group Policy Management > Domain > Group Policy Objects > Default or Active Domain Policy, and then select Edit.

   b. On the Group Policy Management Editor pane, select Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > Security Options, and then configure Network security: Configure Encryption types allowed for Kerberos.

   c. Select the encryption algorithm you want to use when you connect to the KDC server. You can select all the options.

   

   d. Use the Ksetup command to specify the encryption algorithm to be used on the specified realm.

   C:> ksetup /SetEncTypeAttr REALM.COM DES-CBC-CRC DES-CBC-MD5 RC4-HMAC-MD5 AES128-CTS-HMAC-SHA1-96 AES256-CTS-HMAC-SHA1-96
   

4. Create the mapping between the domain account and the Kerberos principal, so that you can use the Kerberos principal in the Windows domain.

   a. Select Administrative tools > Active Directory Users and Computers.

   b. Configure advanced features by selecting View > Advanced Features.

   c. On the Advanced Features pane, right-click the account to which you want to create mappings and, on the Name Mappings pane, select the Kerberos Names tab.

   d. Add a principal from the realm.

   

On the self-hosted integration runtime machine:

• Run the following Ksetup commands to add a realm entry.

  C:> Ksetup /addkdc REALM.COM <your_kdc_server_address>
  C:> ksetup /addhosttorealmmap HDFS-service-FQDN REALM.COM
  

In your data factory or Synapse workspace:

• Configure the HDFS connector by using Windows authentication together with either your domain account or Kerberos principal to connect to the HDFS data source. For configuration details, see the HDFS linked service properties section.

Lookup activity properties

For information about Lookup activity properties, see Lookup activity.

Delete activity properties

For information about Delete activity properties, see Delete activity.

Legacy models

Legacy dataset model

Example:

{
    "name": "HDFSDataset",
    "properties": {
        "type": "FileShare",
        "linkedServiceName":{
            "referenceName": "<HDFS linked service name>",
            "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "folderPath": "folder/subfolder/",
            "fileName": "*",
            "modifiedDatetimeStart": "2018-12-01T05:00:00Z",
            "modifiedDatetimeEnd": "2018-12-01T06:00:00Z",
            "format": {
                "type": "TextFormat",
                "columnDelimiter": ",",
                "rowDelimiter": "\n"
            },
            "compression": {
                "type": "GZip",
                "level": "Optimal"
            }
        }
    }
}

Legacy Copy activity source model

Example: HDFS source in Copy activity using DistCp

"source": {
    "type": "HdfsSource",
    "distcpSettings": {
        "resourceManagerEndpoint": "resourcemanagerendpoint:8088",
        "tempScriptPath": "/usr/hadoop/tempscript",
        "distcpOptions": "-m 100"
    }
}

Related content

For a list of data stores that are supported as sources and sinks by the Copy activity, see supported data stores.