New Feature Evaluation Guide for Windows HPC Server 2008 R2 SP2

Download the Lizard utility that is included in the Microsoft HPC Pack 2008 R2 Tool Pack (lizard_x86.msi).

Run the lizard_x86.msi installer on the head node, and change the default installation path to C:\Lizard\.

Prepare the Linpack input file.

Note
This example uses a small problem size (15000 Ns) to speed up the run time. This will not provide a representative measure of cluster performance.  If you are using fewer than 32 cores, change the number of Qs to match the number of cores. For more information about these parameters, see Lizard Help.

Open Notepad and copy the following text into a new file:

HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out      output file name (if any)
6            device out (6=stdout,7=stderr,file)
1            # of problems sizes (N)
15000 Ns
1            # of NBs
160 NBs
0            PMAP process mapping (0=Row-,1=Column-major)
1            # of process grids (P x Q)  (1xN works best on Eth hub)
1  Ps
32 Qs
16.00         threshold
1            # of panel fact
2 PFACTs (0=left, 1=Crout, 2=Right)
1            # of recursive stopping criterium
4 NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
1            # of recursive panel fact.
2            RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
1            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
0            DEPTHs (>=0)
2            SWAP (0=bin-exch,1=long,2=mix)
160            swapping threshold
0            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
0            Equilibration (0=no,1=yes)
16            memory alignment in double (> 0)

Save the file into the Lizard folder as hpl.dat.

Important
The hpl.dat input file must be in the same folder as xhplmkl.exe.

Create a folder named C:\AzurePkgs.

Package the Lizard files for upload to Azure storage using hpcPack and save the package to the C:\AzurePkgs folder. At an elevated command prompt window type the following command:

hpcpack create c:\AzurePkgs\Lizard.zip c:\Lizard

Note
You do not need an elevated command prompt (run as Administrator) to run hpcpack create, but the hpcpack upload and clusrun commands in upcoming steps do require elevation.

Upload the package to Azure storage by using the following command, where myHeadNode is the name of your head node, and myAzureTemplate is the name of the template that you used to deploy the Azure Nodes:

hpcpack upload Lizard.zip /scheduler:myHeadNode /nodetemplate:myAzureTemplate /relativepath:Lizard

The upload might take a couple of minutes to complete.

Use hpcSync to copy the Lizard files from Azure storage to the Azure Nodes. The files will be extracted to %CCP_PACKAGE_ROOT%\Lizard. Run the following command:

clusrun /nodegroup:AzureNodes hpcsync

Use hpcfwutil to register the Linpack application with the firewall on the Azure Nodes. Run the following command:

clusrun /nodegroup:AzureNodes hpcfwutil register lizard """%CCP_PACKAGE_ROOT%\lizard\xhplmkl.exe"""

Use one of the following methods to ensure that the netmask will allow communication between your MPI processes:

• You can disable the netmask (allow all IP addresses) on your Azure Nodes:
  
  clusrun /nodegroup:AzureNodes setx CCP_MPI_NETMASK=”0.0.0.0/0.0.0.0”
  
  
• You can specify the desired range (or disable) at the job level by using the MPI environment variable MPICH_NETMASK. For example, if your Azure Nodes start with 10.28.x.x:
  
  Job submit /nodegroup:azurenodes /numcores:32 /stdout: %CCP_PACKAGE_ROOT%\lizard\out.txt /workdir: %CCP_PACKAGE_ROOT%\lizard mpiexec –env MPICH_NETMASK 10.28.0.0/255.255.0.0 xhplmkl.exe
  
  

Use job submit to submit the Linkpack job. Run the following command (numcores should be the same as the number you specified for Q in the input file):

Job submit /nodegroup:azurenodes /numcores:32 /stdout:%CCP_PACKAGE_ROOT%\lizard\out.txt /workdir:%CCP_PACKAGE_ROOT%\lizard mpiexec xhplmkl.exe

When the job is complete, collect the output file. In this case, only the Rank 0 process writes to the standard out file, so you only need to collect one file. To identify which node has the output file, run the following command:

Clusrun /nodegroup:azurenodes dir %CCP_PACKAGE_ROOT%\lizard\out.txt

Use HpcFile to download the file from the Azure Node to the head node (where you are running the command from). Run the following command, where AzureCN-000x is the node with the out.txt file, and c:\users\myName\ is the destination folder on the head node:

Hpcfile get /targetnode:AzureCN-000x /file:%CCP_PACKAGE_ROOT%\lizard\out.txt /destFile:c:\users\myName\

Open out.txt in Notepad.

Follow the instructions in Step 4: Create a VHD for Azure VM Nodes in a Windows HPC Cluster to prepare a VHD operating system image, and for the “Install and configure your applications” step, install Microsoft Excel 2010 and any updates to Excel that you require. Do not copy workbooks or any small dependencies that are likely to change frequently directly to the VHD image.

Note
The cluster-side features of HPC Services for Excel are automatically included when you install the Windows HPC Server components on the VHD.

Upload the VHD to your Azure account following the instructions in Step 5: Upload a VHD to Azure from a Windows HPC Cluster.

Create an Azure Node template for VM roles. In HPC Cluster Manager, click Configuration, and then click Node Templates.

Click New, and in the Create Node Template Wizard, select the following options:

1. In Choose Node Template Type, select Windows Azure node template.
   
   
2. In Specify Template Name, type AzureVMExcel.
   
   
3. Provide you subscription and service information.
   
   
4. In Specify Node Role, select VM role.
   
   
5. In the VHD image dropdown list, select the VHD image that you just created.
   
   
6. Configure the remote desktop credentials. This enables you to use the Remote Desktop action in HPC Cluster Manager to connect to the Azure Nodes.
   
   
7. Select the option to configure Azure availability policy manually.
   
   

On the head node, create a folder named C:\AzurePkgs.

Package the workbook for upload to Azure storage by using hpcPack and save the package to the C:\AzurePkgs folder. At an elevated command prompt window type the following command, where “c:\Excel\myWorkbook.xlsb” points to your workbook:

hpcpack create C:\AzurePkgs\myWorkbook.zip c:\Excel\myWorkbook.xlsb

Important

The package name must be the same as the workbook name. If your workbook has dependencies such as other workbooks or DLLs, create a folder that includes the workbook and supporting files, and then package the entire folder. For example: hpcpack create C:\AzurePkgs\myWorkbook.zip c:\Excel\myWorkbookFiles. The workbook must be at the top level of the folder that you are packaging and cannot be contained in a sub-folder. In the example, the workbook would be in “c:\Excel\myWorkbookFiles\myWorkbook.xlsb”.

Note
You do not need an elevated command prompt (run as Administrator) to run hpcpack create, but the hpcpack upload command in an upcoming step does require elevation.

Upload the package to Azure storage, for example type:

hpcpack upload myWorkbook.zip /scheduler:myHeadNode /nodetemplate:AzureVMExcel

In HPC Cluster Manager, in Node Management click, Add Node and use the wizard to specify the VM node template, the number, and the size of the nodes to add. The nodes will appear in the node list in the Not-Deployed state.

Select the Azure nodes, right-click, and then click Start. This action deploys a set of VM role instances in Windows Azure, and can take some time to complete.

When the nodes move from the Provisioning state to the Offline state, right-click the nodes and then click Bring Online.

Install the key storage provider (KSP) on the head node, compute nodes, and workstation nodes. The installer is included in the SP2 download. Run the version that is appropriate for the operating system on each computer: HpcKsp64.msi or HpcKsp86.msi.

You can copy the installers to a shared folder that all on-premises nodes can access and then use the clusrun command to install the KSP on all nodes. For example you can copy the installers to the ccpspooldir share on the head node (\\<headnode>\ccpspooldir) and then run the following command (for 64-bit computers):

clusrun msiexec  /passive /I  \\<headnode>\ccpspooldir\hpcksp_x64.msi

Set the HPC Soft Card authentication policy on the head node by setting the HpcSoftCard cluster property. HpcSoftCard property is set to Disabled by default. If you want users to always use soft card authentication, set the property to Required. If you want users to choose between password or soft card log on, set the property to Allowed.

For example, run HPC PowerShell as an Administrator and type:

Set-HpcClusterProperty –HpcSoftCard:Allowed

Or at an elevated command prompt window, type:

cluscfg setparams “hpcSoftCard=Allowed”

Set the HpcSoftCardTemplate cluster property to specify the certificate template that should be used to generate a soft card credential.

For example, run HPC PowerShell as an Administrator and type:

Set-HpcClusterProperty –HpcSoftCardTemplate:

Or at an elevated command prompt window, type:

cluscfg setparams “hpcSoftCardTemplate=”

You can configure the warning period for soft cards that are nearing their expiration date. By default, this value is set to 5 days. If a user tries to submit a job with less than 5 days before their credentials expire, the job will be rejected. The user will see an error message about the soft card expiration, and will need to generate a new soft card certificate before resubmitting the job. You can configure this value by setting the SoftCardExpirationWarning cluster property.

For example, run HPC PowerShell as an Administrator and type:

Set-HpcClusterProperty –SoftCardExpirationWarning:3

Or at an elevated command prompt window, type:

cluscfg setparams “SoftCardExpirationWarning=3”

Note
To disable expiration warnings, you can set SoftCardExpirationWarning to 0.

Log on to the computer with your Smart Card.

Use one of the following methods to generate an HPC Soft Card.

Run HPC PowerShell and type:

New-HpcSoftCard

Or at a command prompt window, type:

hpccred createcert

Use one of the following methods to delete any previously cached credentials, if any:

Run HPC PowerShell and type:

remove-hpcJobCredential

Or at a command prompt window, type:

hpccred delcreds

Submit a test job, for example:

Run HPC PowerShell and type:

New-hpcJob|add-hpcTask –command:”echo hello”|submit-hpcjob

Or at a command prompt window, type:

Job submit echo hello

When prompted, select which credentials to use.

In HPC Cluster Manager, click Configuration.

In the Navigation Pane, click Resource Pools.

In Actions, click Edit Pools and Weights. The dialog box appears.

In the dialog box, click Add two times. Two new rows appear in the list of pools.

In the Pool Name column, rename the pools PoolA and PoolB.

In the Weight column, type the desired weights for each group (60, 40).

Set the weight for the Default pool to 0.

Click OK to save and close the dialog box.

In the Navigation Pane, click Job Templates.

In Actions, click New to open the Job Template wizard and then define the template as follows:

• Name: GroupA_HighPriJobs
  
  
• Maximum priority: Highest
  
  
• Default priority: Highest
  
  

Open the job template “GroupA_HighPriJobs” in the Job Template Editor and define the Pool and the user permissions as follows:

• Add the Pool property to the template, and for the Valid Value, select “PoolA”.
  
  
• Click Permissions, and ensure that only users in Group A have permission to submit jobs with that template.
  
  
• Save the changes.
  
  

Create a job template for “GroupA_LowPriJobs” as follows:

• In the Job Template list, right-click “GroupA_HighPriJobs” and then click Copy.
  
  
• Right-click the copy, click Edit, and then define the template as follows:
  
  
• Name: GroupA_LowPriJobs
  
  
• Maximum priority: Normal
  
  
• Default priority: BelowNormal
  
  
• Because this template is based on the “GroupA_HighPriJobs”, the permissions and Pool are already set correctly.
  
  

Create a new job template for “GroupB_AllJobs” with the following properties:

• Name: GroupB_AllJobs
  
  
• Maximum priority: Highest
  
  
• Default priority: Normal
  
  
• Add the Pool property to the template, and for the Valid Value, select “PoolB”.
  
  
• Click Permissions, and ensure that only users in Group B have permission to submit jobs with that template.
  
  

In HPC Cluster Manager, click Options, and then click Job Scheduler Configuration.

Select the Resource Pools tab.

Select the Enable Resource Pools check box.

Click OK.

Run HPC PowerShell as an Administrator.

List node names, core, and socket counts:

get-hpcnode|select netbiosname, processorcores, sockets, subscribedcores, subscribedsockets

Take CN001 Offline (if you want to cancel any tasks that are running on the node, include the –force parameter):

set-hpcnodestate –name:CN001 –state:offline

Set subscribed cores and sockets on the node:

set-hpcnode –name:CN001 –subscribedcores:8 –subscribedsockets:2

Verify the property changes:

get-hpcnode –name:CN001|select netbiosname, processorcores, sockets, subscribedcores, subscribedsockets

Note
Node properties changes are applied during scheduling passes. If your changes are not reflected when you run get-hpcnode, wait a few seconds and then try again.

Bring the node Online:

set-hpcnodestate –name:CN001 –state:online

In HPC Cluster Manager, click Node Management.

Click the blank tab to create a new node view, and then click Customize Tab.

In the Customize Tab dialog box, select the following columns:

• CPU Usage (%)
  
  
• Cores In Use (Note: this reflects physical cores, not subscribed cores)
  
  
• Subscribed Cores
  
  
• Subscribed Sockets
  
  
• Affinity
  
  
• Running Tasks
  
  
• Cores
  
  
• Sockets
  
  

You can sort the list by the value of Subscribed Cores by clicking the Subscribed Cores column header in the list view.

To see the running jobs for a node, select the node in the node list, and then in the Actions Pane, click Jobs for the Selected Nodes.

On the head node, run HpcWebComponents.msi.

Run HPC PowerShell as an Administrator, and then type:

Powershell.exe –executionpolicy bypass –command set-hpcwebcomponents portal -enable

Select a certificate option from the displayed list. For testing purposes you can type 0 to generate and configure a self-signed certificate.

When configuration and installation are complete, open Internet Explorer and add the web portal to the list of trusted sites as follows:

1. On the Tools menu, click Internet Options.
   
   
2. In the Security tab, select the Trusted sites zone, and then click Sites.
   
   
3. Add https://localhost/hpcportal to the list of sites.
   
   

   Note
   The default security level for trusted sites (Medium) allows AJAX, which is required to view the web portal.

Open Internet Explorer and go to the following address:

https://localhost/hpcportal

• If you see a certificate error warning, click “Continue to this website”.
  
  
• If you are prompted, type your doMayn credentials.
  
  

In the Windows HPC Server Web Portal, click Application Profiles.

Click New.

Define the computer name parameter for the ping command by specifying the following properties:

• Type: Text
  
  
• Name: computer
  
  
• Default value: localhost
  
  
• Format: {1}
  
  

Click Add to specify a second parameter.

Define the count parameter for the ping command by specifying the following properties:

• Type: Numeric
  
  
• Name: count
  
  
• Default value: 2
  
  
• Switch: n
  
  
• Format: -{0} {1}
  
  

In Specify a command, type ping.

Click Show preview and verify that the command appears as expected. In this example, the command appears as:

ping localhost –n 2

In Save this application profile as, type “PingCommand”, and then click Save.

In the Windows HPC Server Web Portal, click Submission pages, and then click New.

In the name and job template page, specify the following properties:

1. Name for submission page: “PingJob”
   
   
2. Job template: “Default”
   
   
3. Page type: “Parametric Sweep Job”
   
   

In the job property visibility and defaults page, configure the properties as follows:

1. Job Name: “myPingJob”.
   
   
2. Project Name: Clear the Show check box for this property.
   
   
3. Define the parametric sweep values: start=1, end=50, increment=1.
   
   
4. Clear the Show check boxes for the eMayl, working directory, and standard in, out and error properties.
   
   

On the next page, define Node Preparation and Release tasks as follows:

1. Do not show the Node Preparation task options, and specify a default value of “echo hello”.
   
   

   Note

   If a property is not shown, the default value is applied, and the job owner cannot change the value from the job submission page. After the job is submitted, the job owner will be able to see the values for all properties, and modify values within the restrictions defined by the underlying job template, as permitted by the job state. For information about what properties can be modified in different job states, see Modify a Job.

2. Select the check box to show the Node Release task options, and specify a default value of “echo goodbye”.
   
   

On the Specify application profile page, select the option to use an existing profile, click Select, select “PingCommand” in the dialog box, and then click OK.

On the next page, accept the default visibility settings for the PingCommand Application Profile.

On the last page, click Finish.

In the Windows HPC Server Web Portal, click New Job, and then click “PingJob”.

In the PingJob submission page, under Application parameters, verify that the following default values appear:

1. In “computer”, type “localhost”.
   
   
2. In “count”, type “2”.
   
   

In the Node Prep/Node Release Tasks section, notice that the Node Preparation task (echo hello) is not displayed, but you can modify the default command for the Node Release task (echo goodbye).

Click Submit.

In the job list, right-click “myPingJob”, click View Job, and then select the View Tasks tab.

On your development computer, extract the HPC2008R2.SampleCode.zip file.

In the sample code folder, open the SubmissionJobSize folder (in the HPC2008R2.SampleCode\Scheduler\Filters\dll folder).

Open the SubmissionJobSize Visual C# Project file with Visual Studio.

In the Build menu, click Build Solution to compile the sample, as illustrated in the following screen shot:

Accept the default save location.

The SubmissionJobSize folder now includes a \bin\Debug folder. The Release folder includes the SubmissionJobSize DLL (application extension file) and PDB (program debug database file).

On the head node, open the %CCP_DATA%Filters folder (typically, this is C:\Program Files\Microsoft HPC Pack 2008 R2\Data\Filters).

Create a new sub-folder named “SubmissionJobSize”.

Copy the SubmissionJobSize DLL and PDB files (from the SubmissionJobSize\bin\Debug folder) and place them in the new folder you just created.

Note
If your filters have more than one file, or if they create output files, it is good practice to create a sub-folder for each filter in the %CCP_DATA%Filters folder. The PDB file is not required, but you can include it to help when debugging the filter.

In HPC Cluster Manager, in Configuration, select Job Templates.

Click New.

In the wizard, type “LargeJobTemplate” for the template name, select the Finish tab, and then click Finish.

Right-click the Default job template and then click Edit.

Click Add, and then select SubmissionFilters.

In Valid Values, specify the location of the filter relative to the %CCP_DATA%Filters folder. In this case, type SubmissionJobSize\SubmissionJobSize.dll, as illustrated in the following screen shot:

Note
You can specify more than one filter in the value field. List each filter on its own line (filters are delimited by a carriage return). Filters run in the listed order, and will run before the cluster-wide filter, if specified.

Click Save.

In HPC Cluster Manager, click Job Management.

Click New Job and define the job as follows:

1. Job name: FilterTest
   
   
2. Job template: Default
   
   
3. Resource type: Core
   
   
4. Maximum: 2
   
   

Click Edit Tasks, click Add, and then select Basic Task.

In command line, type echo hello, and then click OK.

Click Submit.

In the job list, select your “FilterTest” job, and then in the Details pane, click the Job Details tab to view the assigned job template.