hpcsync

Copies OPC packages from a Windows Azure storage account and unzips them on a Windows Azure node.

Syntax

hpcsync [<storageAccountName> <storageKey>] [<packageRoot>] [/packageName:<packageNameFilter>]   
[/container:<Container name>] [/blocksize:<blockSize>] [/usehttp]   
  
hpcsync {/? | /help}  

Parameters

In some cases, a cluster administrator can run clusrun hpcsync without specifying any parameters for the hpcsync command.

Remarks

• hpcsync is available on Windows Azure compute resources that are joined to an on-premises cluster that is running at least Microsoft HPC Pack 2008 R2 with SP1, or Windows Azure compute resources that are part of a Windows Azure HPC Scheduler deployment.

• hpcsync runs automatically when you start Windows Azure nodes from HPC Cluster Manager, and it also runs automatically if any of the node instances are reprovisioned as part of Windows Azure maintenance. This mechanism helps ensure that required files are available on the nodes at startup or if the nodes instances are reprovisioned. The automatic hpcsync calls at node provisioning run without any parameters. This means that packages are copied from the default location in storage to the default location on the nodes. For more information, see Leveraging automatic hpcsync calls for application deployment later in this topic.

• Because it runs automatically during burst to Windows Azure deployments, you do not necessarily need to run hpcsync directly. However, if you stage packages to storage after the Windows Azure nodes are already deployed, or if you are deploying packages to a Windows Azure HPC Scheduler deployment, you can manually run hpcsync with clusrun. hpcsync also provides some parameters for copying packages from different storage containers, and to deploy the packages to alternate target directories.

• hpcsync is ideal for deploying SOA services, XLL files, and startup scripts from Windows Azure storage to their expected locations on your Windows Azure nodes. Alternatively, you can use the hpcpack commands (hpcpack upload and hpcpack download) to move other types of files, applications, or job data between Windows Azure storage and your Windows Azure nodes. For more information, see hpcpack.

• To use this command, cluster users must know the storage account name and primary access key.

Examples

For a cluster administrator to deploy all OPC files from the default container on a Windows Azure storage account to a set of Windows Azure worker nodes:

clusrun /nodegroup:AzureWorkerNodes hpcSync

For a cluster user to deploy all OPC files from a specific container on a Windows Azure storage account (myContainer) to their default directory on a set of Windows Azure nodes (nodes 1-3):

job submit /requestednodes:node1,node2,node3 hpcSync myAccount myKey /container:myContainer

To help troubleshoot file deployment, you can test the command by trying to deploy the files to your local computer in C:\temp:

hpcsync myAccount myKey C:\temp

Leveraging automatic hpcsync calls for application deployment

To leverage the automatic hpcsync calls during node provisioning, you must stage OPC packages to the hpcpackages container on the Windows Azure storage account before deploying your nodes. You can use the hpcpack commands to package and then upload files to Windows Azure storage. (By default, hpcpack upload places packages in the hpcpackages container; you specify an alternative container if you do not want the packages to be automatically deployed to all Windows Azure nodes.)

By default, for cluster administrators hpcsync deploys files to a location on the Windows Azure nodes that is determined in part by the %CCP_PACKAGE_ROOT% environment variable. This variable is set on Windows Azure nodes during the provisioning process. The extracted files are placed in a folder that is determined as follows: %CCP_PACKAGE_ROOT%\<packageName>\<uploadTimeStamp>. This is the expected location for SOA services, XLLs, Excel workbooks, and startup scripts that are called from the node template. However, this is not convenient for applications that cluster users will be calling in their command lines.

To simplify the folder structure for executable files, you can set the relative path property for the package when you upload it to storage. hpcsync applies the relative path when extracting the files, so that the path is determined as follows: %CCP_PACKAGE_ROOT%\<relativePath>. Users can then specify the path to their application as in the following example of a job submit command: job submit %CCP_PACKAGE_ROOT%\myRelativePath\myapp.exe

For more information, see Deploying Applications to Windows Azure Nodes in a Windows HPC Cluster.

Important considerations about %CCP_PACKAGE_ROOT%

• On Windows Azure nodes, the %CCP_PACKAGE_ROOT% folder is created on a 10 GB disk partition. This means that all application files on a node instance cannot exceed 10 GB. If an application has considerable input and output files, you can use a startup script to grant user permissions on the C:\ drives so that users can write to all available scratch space on the node.

• When you run hpcsync manually, you can override the default location (%CCP_PACKAGE_ROOT%). For example, you can create a folder on each Windows Azure node and then specify that location when you run hpcsync. All packages will be extracted to that folder. However, any new node instances that are deployed (or automatically reprovisioned) will not include that folder, and packages will be automatically deployed to the default location.

• Cluster users only have write permissions to folders in %CCP_PACKAGE_ROOT%. Unless you modify folder permissions on the Windows Azure nodes, only cluster administrators can run applications outside of %CCP_PACKAGE_ROOT%.

• When hpcsync deploys a package, none of the extracted files can have a full path over 256 characters. The root directories where the extracted files are temporarily and then finally placed can take up to 136 characters, leaving 120 characters for the file name, subdirectories (if any), and the relativePath (if specified). If the path for the extracted files exceeds 256 characters, the package deployment fails.

See Also

hpcpack
clusrun
Burst to Azure Worker Instances with Microsoft HPC Pack