Connect to HDInsight (Apache Hadoop) using SSH

Learn how to use Secure Shell (SSH) to securely connect to Apache Hadoop on Azure HDInsight. For information on connecting through a virtual network, see Azure HDInsight virtual network architecture. See also, Plan a virtual network deployment for Azure HDInsight clusters.

The following table contains the address and port information needed when connecting to HDInsight using an SSH client:

Replace <clustername> with the name of your cluster. Replace <edgenodename> with the name of the edge node.

If your cluster contains an edge node, we recommend that you always connect to the edge node using SSH. The head nodes host services that are critical to the health of Hadoop. The edge node runs only what you put on it. For more information on using edge nodes, see Use edge nodes in HDInsight.

SSH clients

Linux, Unix, and macOS systems provide the ssh and scp commands. The ssh client is commonly used to create a remote command-line session with a Linux or Unix-based system. The scp client is used to securely copy files between your client and the remote system.

Microsoft Windows doesn't install any SSH clients by default. The ssh and scp clients are available for Windows through the following packages:

• OpenSSH Client. This client is an optional feature introduced in the Windows 10 Fall Creators Update.

• Bash on Ubuntu on Windows 10.

• Azure Cloud Shell. The Cloud Shell provides a Bash environment in your browser.

• Git.

There are also several graphical SSH clients, such as PuTTY and MobaXterm. While these clients can be used to connect to HDInsight, the process of connecting is different than using the ssh utility. For more information, see the documentation of the graphical client you're using.

Authentication: SSH Keys

SSH keys use public-key cryptography to authenticate SSH sessions. SSH keys are more secure than passwords, and provide an easy way to secure access to your Hadoop cluster.

If your SSH account is secured using a key, the client must provide the matching private key when you connect:

• Most clients can be configured to use a default key. For example, the ssh client looks for a private key at ~/.ssh/id_rsa on Linux and Unix environments.

• You can specify the path to a private key. With the ssh client, the -i parameter is used to specify the path to private key. For example, ssh -i ~/.ssh/id_rsa [email protected].

• If you have multiple private keys for use with different servers, consider using a utility such as ssh-agent (https://en.wikipedia.org/wiki/Ssh-agent). The ssh-agent utility can be used to automatically select the key to use when establishing an SSH session.

Create an SSH key pair

Use the ssh-keygen command to create public and private key files. The following command generates a 2048-bit RSA key pair that can be used with HDInsight:

ssh-keygen -t rsa -b 2048

You're prompted for information during the key creation process. For example, where the keys are stored or whether to use a passphrase. After the process completes, two files are created; a public key and a private key.

• The public key is used to create an HDInsight cluster. The public key has an extension of .pub.

• The private key is used to authenticate your client to the HDInsight cluster.

Create HDInsight using the public key

Creation method	How to use the public key
Azure portal	Uncheck Use cluster login password for SSH, and then select Public Key as the SSH authentication type. Finally, select the public key file or paste the text contents of the file in the SSH public key field.
Azure PowerShell	Use the -SshPublicKey parameter of the New-AzHdinsightCluster cmdlet and pass the contents of the public key as a string.
Azure CLI	Use the --sshPublicKey parameter of the az hdinsight create command and pass the contents of the public key as a string.
Resource Manager Template	For an example of using SSH keys with a template, see Deploy HDInsight on Linux with SSH key. The publicKeys element in the azuredeploy.json file is used to pass the keys to Azure when creating the cluster.

Authentication: Password

SSH accounts can be secured using a password. When you connect to HDInsight using SSH, you're prompted to enter the password.

Create HDInsight using a password

Creation method	How to specify the password
Azure portal	By default, the SSH user account has the same password as the cluster login account. To use a different password, uncheck Use cluster login password for SSH, and then enter the password in the SSH password field.
Azure PowerShell	Use the --SshCredential parameter of the New-AzHdinsightCluster cmdlet and pass a PSCredential object that contains the SSH user account name and password.
Azure CLI	Use the --ssh-password parameter of the az hdinsight create command and provide the password value.
Resource Manager Template	For an example of using a password with a template, see Deploy HDInsight on Linux with SSH password. The linuxOperatingSystemProfile element in the azuredeploy.json file is used to pass the SSH account name and password to Azure when creating the cluster.

Change the SSH password

For information on changing the SSH user account password, see the Change passwords section of the Manage HDInsight document.

Authentication domain joined HDInsight

If you're using a domain-joined HDInsight cluster, you must use the kinit command after connecting with SSH local user. This command prompts you for a domain user and password, and authenticates your session with the Microsoft Entra domain associated with the cluster.

You can also enable Kerberos Authentication on each domain joined node (for example, head node, edge node) to ssh using the domain account. To do this edit sshd config file:

sudo vi /etc/ssh/sshd_config

uncomment and change KerberosAuthentication to yes

sudo service sshd restart

Use klist command to verify whether the Kerberos authentication was successful.

For more information, see Configure domain-joined HDInsight.

Connect to nodes

The head nodes and edge node (if there's one) can be accessed over the internet on ports 22 and 23.

• When connecting to the head nodes, use port 22 to connect to the primary head node and port 23 to connect to the secondary head node. The fully qualified domain name to use is clustername-ssh.azurehdinsight.net, where clustername is the name of your cluster.

  # Connect to primary head node
  # port not specified since 22 is the default
  ssh [email protected]
  
  # Connect to secondary head node
  ssh -p 23 [email protected]
  

• When connecting to the edge node, use port 22. The fully qualified domain name is edgenodename.clustername-ssh.azurehdinsight.net, where edgenodename is a name you provided when creating the edge node. clustername is the name of the cluster.

  # Connect to edge node
  ssh [email protected]
  

Once connected, the prompt changes to indicate the SSH user name and the node you're connected to. For example, when connected to the primary head node as sshuser, the prompt is sshuser@<active-headnode-name>:~$.

Connect to worker and Apache Zookeeper nodes

The worker nodes and Zookeeper nodes aren't directly accessible from the internet. They can be accessed from the cluster head nodes or edge nodes. The following are the general steps to connect to other nodes:

1. Use SSH to connect to a head or edge node:

   ssh [email protected]
   

2. From the SSH connection to the head or edge node, use the ssh command to connect to a worker node in the cluster:

   ssh sshuser@wn0-myhdi
   

   To retrieve a list of the node names, see the Manage HDInsight by using the Apache Ambari REST API document.

If the SSH account is secured using a password, enter the password when connecting.

If the SSH account is secured using SSH keys, make sure that SSH forwarding is enabled on the client.

Configure SSH agent forwarding

1. Using a text editor, open ~/.ssh/config. If this file doesn't exist, you can create it by entering touch ~/.ssh/config at a command line.

2. Add the following text to the config file.

   Host <edgenodename>.<clustername>-ssh.azurehdinsight.net
       ForwardAgent yes
   

   Replace the Host information with the address of the node you connect to using SSH. The previous example uses the edge node. This entry configures SSH agent forwarding for the specified node.

3. Test SSH agent forwarding by using the following command from the terminal:

   echo "$SSH_AUTH_SOCK"
   

   This command returns information similar to the following text:

   /tmp/ssh-rfSUL1ldCldQ/agent.1792
   

   If nothing is returned, then ssh-agent isn't running.

4. Once you've verified that ssh-agent is running, use the following to add your SSH private key to the agent:

   ssh-add ~/.ssh/id_rsa
   

   If your private key is stored in a different file, replace ~/.ssh/id_rsa with the path to the file.

5. Connect to the cluster edge node or head nodes using SSH. Then use the SSH command to connect to a worker or zookeeper node. The connection is established using the forwarded key.

Next steps

• Use SSH tunneling with HDInsight
• Use edge nodes in HDInsight
• Use SCP with HDInsight