Operating in a locked down network

The CycleCloud application and cluster nodes can operate in environments with limited internet access, though you must keep a minimal number of TCP ports open.

Install Azure CycleCloud in a locked down network

The CycleCloud VM needs to connect to several Azure APIs to manage cluster VMs and authenticate to Azure Active Directory. Since these APIs use HTTPS, CycleCloud requires outbound HTTPS access to:

• management.azure.com (Azure ARM Management)
• login.microsoftonline.com (Azure AD)
• watson.telemetry.microsoft.com (Azure Telemetry)
• dc.applicationinsights.azure.com (Azure Application Insights)
• dc.applicationinsights.microsoft.com (Azure Application Insights)
• dc.services.visualstudio.com (Azure Application Insights)
• ratecard.azure-api.net (Azure Price Data)

The management API is hosted regionally. You can find the public IP address ranges here.

The Azure AD authentication is part of the Microsoft 365 common APIs. You can find the IP address ranges for this service here.

You can find the IP address ranges for Azure Insights and Log Analytics here.

Azure CycleCloud must access Azure Storage accounts. To provide private access to this service and any other supported Azure service, we recommend using Virtual Network Service Endpoints.

If you use Network Security Groups or the Azure Firewall to limit outbound access to the required domains, you can configure Azure CycleCloud to route all requests through an HTTPS proxy. For more information, see Using a Web Proxy.

Configuring an Azure Network Security Group for the CycleCloud VM

You can limit outbound internet access from the CycleCloud VM by configuring a strict Azure Network Security Group for the CycleCloud VM's subnet. This approach doesn't require configuring the Azure Firewall or an HTTPS proxy. The simplest way to do that is to use Service Tags in the subnet or VM level Network Security Group to permit the required outbound Azure access.

1. Configure a Storage Service Endpoint for the Subnet to allow access from CycleCloud to Azure Storage

2. Add the following NSG Outbound rule to Deny outbound access by default using the "Internet" destination Service Tag:

   Priority	Name	Port	Protocol	Source	Destination	Action
   4000	BlockOutbound	Any	Any	Any	Internet	Deny

3. Add the following NSG Outbound rules to Allow outbound access to the required Azure services by destination Service Tag:

   Priority	Name	Port	Protocol	Source	Destination	Action
   100	AllowAzureStorage	443	TCP	Any	Storage	Allow
   101	AllowActiveDirectory	443	TCP	Any	AzureActiveDirectory	Allow
   102	AllowAzureMonitor	443	TCP	Any	AzureMonitor	Allow
   103	AllowAzureRM	443	TCP	Any	AzureResourceManager	Allow

Internal communications between cluster nodes and CycleCloud

Open these ports to allow communication between the cluster nodes and CycleCloud server:

Launching Azure CycleCloud clusters in a locked down network

Running VMs or Cyclecloud clusters in a virtual network or subnet with outbound internet access generally requires the following steps:

1. Make Azure Cyclecloud reachable from the cluster VMs for full functionality. Either:
   1. Cluster VMs connect to Azure Cyclecloud directly via HTTPS and AMQP, or
   2. Enable the Cyclecloud ReturnProxy feature when you create the cluster. Cyclecloud must be able to connect to the ReturnProxy VM through SSH.
2. Make sure the cluster VMs have all the required software packages by:
   1. Preinstalling them in a custom Managed Image,
   2. Providing a package repository mirror that the VMs can access, or
   3. Copying them to the VM from Azure Storage and installing them directly through a Cyclecloud project.
3. Make sure all cluster nodes can access Azure Storage accounts. To provide private access to this service and any other supported Azure service, enable a Virtual Network Service Endpoint for Azure Storage.

Project updates from GitHub

CycleCloud downloads cluster projects from GitHub during the Staging orchestration phase. This download happens after initial installation, after upgrading CycleCloud, or when you start a cluster of a certain type for the first time. In a locked down environment, HTTPS outbound traffic to github.com might be blocked. If this traffic is blocked, node creation during the staging resources phase fails.

If you can temporarily open access to GitHub during the creation of the first node, CycleCloud prepares the local files for all subsequent nodes. If temporary access isn't possible, you can download the necessary files from another machine and copy them to CycleCloud.

First, determine which project and version your cluster needs, such as Slurm 3.0.8. It's usually the highest version number in the database for a given project. You can find the latest version by visiting the GitHub project page or by querying CycleCloud for the latest version.

To query CycleCloud (note that there are often multiple versions listed):

/opt/cycle_server/cycle_server execute 'select Name, Version, Url from cloud.project where name == "slurm" order by Version'

Name = "slurm"
Version = "3.0.8"
Url = "https://github.com/Azure/cyclecloud-slurm/releases/3.0.8"

You can find this project version and all dependencies in the release tag.

You can manually download all release artifacts, but the CycleCloud CLI provides a helper for this operation.

First, use the CycleCloud CLI to fetch and prepare the repository from GitHub. This operation is the same operation CycleCloud performs during the "Staging Resources" phase:

RELEASE_URL="https://github.com/Azure/cyclecloud-slurm/releases/3.0.8"
RELEASE_VERSION="3.0.8"
mkdir "${RELEASE_VERSION}"
cd "${RELEASE_VERSION}"
# Download release artifacts from githug (on a machine with github access)
cyclecloud project fetch "${RELEASE_URL}" .

# Create a tarball with the project files pre-staged
cyclecloud project build
mv ./build/slurm "./${RELEASE_VERSION}"
tar czf "slurm-${RELEASE_VERSION}.tgz" ./blobs "./${RELEASE_VERSION}"

Next, copy the packaged project tarball to the CycleCloud server and extract it:

#... copy the "slurm-${RELEASE_VERSION}.tgz" file to the Cyclecloud server in /tmp
sudo -i
mkdir -p /opt/cycle_server/work/staging/projects/slurm
cd /opt/cycle_server/work/staging/projects/slurm
tar xzf "/tmp/slurm-${RELEASE_VERSION}.tgz"
chown -R cycle_server:cycle_server /opt/cycle_server/work/staging

Once you stage these files locally, CycleCloud detects them and doesn't try to download them from GitHub.