Configure and use cross-cluster networking for Azure Kubernetes Fleet Manager (preview)

Applies to: ✔️ Fleet Manager ✔️ Fleet Manager with hub cluster

Cross-cluster networking for Azure Kubernetes Fleet Manager provides managed Cilium multi-cluster that extends Cilium's eBPF-based networking and observability across multiple clusters.

Fleet Manager supports creation of multiple cross-cluster networking profiles with each supporting up to 255 member clusters. Clusters in a profile participate in a managed federated network with access to global services and service discovery and observability via Hubble.

Platform engineers label member clusters that should participate in a single cross-cluster network, specifying the label in a selector when defining the network profile. To allow for bulk membership management and avoid unintended interruptions, the cross-cluster network is only updated when explicitly requested by an authorized user.

This article provides instructions on how to configure a cross-cluster networking profile, add member clusters and how to expose global services.

Prerequisites and limitations

• A cross-cluster network can have up to 255 member clusters.
• Fleet Manager member clusters can only participate in a single cross-cluster network at any time.
• Clusters must run Kubernetes v1.32 or later and have Advanced Container Networking Services (ACNS) with Cilium enabled.
• Clusters must be connected to a single flat network (virtual network or multiple peered networks).
• Overlay networking with tunnels isn't currently supported.
• Self-managed Cilium multi-cluster can't be deployed at the same time.
• ACNS sets the Cilium version and enabled features which are read-only and can't be modified by users.

Before you begin

If you don't have an Azure account, create a free account before you begin.

Install the Cilium CLI so you can verify the cross-cluster network in a later step.

Set up your environment

If you're not currently using Fleet Manager and want to try out cross-cluster networking, follow the steps in this section.

If you already have a configured environment, you can skip to Create a cross-cluster network.

Set the following environment variables:

export CLUSTER_RESOURCE_GROUP=<cluster-resource-group>
export FLEET_RESOURCE_GROUP=<fleet-resource-group>
export RESOURCE_LOCATION=<azure-region-name>
export FLEET=<fleet-name>
export MEMBER_CLUSTER_1=aks-member-1
export MEMBER_CLUSTER_2=aks-member-2
export NETWORK_NAME=<network-name>
export FLEET_NETWORK_PROFILE_NAME=fccnp-demo-01
export CC_NETWORK_NAME=demo-01

Create network and subnets

Create a new Azure Virtual Network.

az network vnet create \
    --resource-group ${CLUSTER_RESOURCE_GROUP} \
    --location ${RESOURCE_LOCATION} \
    --name ${NETWORK_NAME} \
    --address-prefixes 10.0.0.0/8 \
    --output none

Then create four subnets that you can use to connect the two clusters.

# Subnets for cluster 1

SUBNET_ID_1=$(az network vnet subnet create \
    --resource-group ${CLUSTER_RESOURCE_GROUP} \
    --vnet-name ${NETWORK_NAME} \
    --name ${NETWORK_NAME}-01 \
    --address-prefixes 10.240.0.0/16 \
    --output tsv \
    --query id)

SUBNET_ID_2=$(az network vnet subnet create \
    --resource-group ${CLUSTER_RESOURCE_GROUP} \
    --vnet-name ${NETWORK_NAME} \
    --name ${NETWORK_NAME}-02 \
    --address-prefixes 10.241.0.0/16 \
    --output tsv \
    --query id)

# Subnets for cluster 2

SUBNET_ID_3=$(az network vnet subnet create \
    --resource-group ${CLUSTER_RESOURCE_GROUP} \
    --vnet-name ${NETWORK_NAME} \
    --name ${NETWORK_NAME}-03 \
    --address-prefixes 10.242.0.0/16 \
    --output tsv \
    --query id)

SUBNET_ID_4=$(az network vnet subnet create \
    --resource-group ${CLUSTER_RESOURCE_GROUP} \
    --vnet-name ${NETWORK_NAME} \
    --name ${NETWORK_NAME}-04 \
    --address-prefixes 10.243.0.0/16 \
    --output tsv \
    --query id)

Create AKS clusters

Create two AKS clusters with ACNS enabled, using a flat network such as the one you created in the previous section.

For the first cluster, select the appropriate subnets.

    CLUSTER_ID_1=$(az aks create \
        --name ${MEMBER_CLUSTER_1} \
        --resource-group ${CLUSTER_RESOURCE_GROUP} \
        --location ${RESOURCE_LOCATION} \
        --node-count 1 \
        --vnet-subnet-id ${SUBNET_ID_1} \
        --pod-subnet-id ${SUBNET_ID_2} \
        --network-plugin azure \
        --network-dataplane cilium \
        --enable-acns \
        --generate-ssh-keys \
        --output tsv \
        --query id)

Use the remaining subnets for the second cluster as shown.

CLUSTER_ID_2=$(az aks create \
    --name ${MEMBER_CLUSTER_2} \
    --resource-group ${CLUSTER_RESOURCE_GROUP} \
    --location ${RESOURCE_LOCATION} \
    --node-count 1 \
    --vnet-subnet-id ${SUBNET_ID_3} \
    --pod-subnet-id ${SUBNET_ID_4} \
    --network-plugin azure \
    --network-dataplane cilium \
    --enable-acns \
    --generate-ssh-keys \
    --output tsv \
    --query id)

Create a Fleet Manager and join clusters

In this guide, you create a new Fleet Manager. You don't need a hub cluster.

az fleet create \
    --name ${FLEET} \
    --resource-group ${FLEET_RESOURCE_GROUP} \
    --location ${RESOURCE_LOCATION} 

az fleet member create \
    --fleet-name ${FLEET} \
    --resource-group ${FLEET_RESOURCE_GROUP} \
    --name mbr-${MEMBER_CLUSTER_1} \
    --member-cluster-id ${CLUSTER_ID_1} \
    --labels "network=${CC_NETWORK_NAME}"

az fleet member create \
    --fleet-name ${FLEET} \
    --resource-group ${FLEET_RESOURCE_GROUP} \
    --name mbr-${MEMBER_CLUSTER_2} \
    --member-cluster-id ${CLUSTER_ID_2} \
    --labels "network=${CC_NETWORK_NAME}"

Create a cross-cluster network

Fleet Manager supports multiple cross-cluster networks, with each network configuration defined in a cross-cluster network profile.

Let's set up a new cross-cluster network by using the AKS clusters you provisioned.

Confirm cross-cluster network connection

When the member cluster states show as Connected, the cross-cluster network is established.

1. Get the Kubernetes access credentials for both member clusters by using az aks get-credentials and set context appropriately.

   az aks get-credentials \
       --resource-group ${GROUP} \
       --name ${MEMBER_CLUSTER_1} \
       --context cluster1
   

   az aks get-credentials \
       --resource-group ${GROUP} \
       --name ${MEMBER_CLUSTER_2} \
       --context cluster2
   

2. Use the Cilium CLI status command to see that all member clusters are connected:

   cilium clustermesh status --context cluster1
   

   ✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
   ✅ Cluster access information is available:
     - 10.240.0.5:2379
   ✅ Deployment clustermesh-apiserver is ready
   ℹ️  KVStoreMesh is enabled
   
   ✅ All 1 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
   ✅ All 1 KVStoreMesh replicas are connected to all clusters [min:1 / avg:1.0 / max:1]
   
   🔌 Cluster Connections:
     - mbr-aks-member-021: 1/1 configured, 1/1 connected - KVStoreMesh: 1/1 configured, 1/1 connected
   

   Note

   Fleet Manager cross-cluster networking appends the Cilium cluster ID to the Fleet Manager member cluster name when registering it with the network. This registration adds a number to the peer in Cluster Connections. For example, the peer appears as <fleet-member-name> followed by a number, such as mbr-aks-member-22.

Setup and test global services

After you create the cross-cluster network, test load balancing by following the official Cilium multi-cluster example or using the steps shown next. The steps in this article provide extra guidance on working with AKS clusters and Fleet Manager.

• On mbr-aks-member-1 create a dedicated namespace and annotate it so Services within it are eligible to be shared across the cross-cluster network. Then apply the Deployment and Service resources using the cluster1.yaml manifest.

  kubectl --context=cluster1 create namespace rebel-base-demo
  kubectl --context=cluster1 annotate namespace rebel-base-demo clustermesh.cilium.io/global="true" --overwrite
  kubectl --context=cluster1 -n rebel-base-demo apply -f https://raw.githubusercontent.com/cilium/cilium/refs/heads/main/examples/kubernetes/clustermesh/cluster1.yaml
  kubectl --context=cluster1 -n rebel-base-demo apply -f https://raw.githubusercontent.com/cilium/cilium/refs/heads/main/examples/kubernetes/clustermesh/global-service-example.yaml
  

  Note

  The example Cilium manifests use container images from public registries (docker.io and quay.io). If your environment uses Azure Policy or similar admission controls to restrict container image sources, the example Deployments might generate policy warnings or be blocked. In that case, mirror the images to a registry that is allowed by your policy and update the manifests accordingly.

• On mbr-aks-member-2 apply the equivalent, but this time use the cluster2.yaml manifest.

  kubectl --context=cluster2 create namespace rebel-base-demo
  kubectl --context=cluster2 annotate namespace rebel-base-demo clustermesh.cilium.io/global="true" --overwrite
  kubectl --context=cluster2 -n rebel-base-demo apply -f https://raw.githubusercontent.com/cilium/cilium/refs/heads/main/examples/kubernetes/clustermesh/cluster2.yaml
  kubectl --context=cluster2 -n rebel-base-demo apply -f https://raw.githubusercontent.com/cilium/cilium/refs/heads/main/examples/kubernetes/clustermesh/global-service-example.yaml
  

• Run the following command multiple times on each cluster. Observe the serving cluster changes, demonstrating the request is being served by multiple clusters despite calling the service on only one.

  kubectl --context=cluster1 -n rebel-base-demo exec -ti deployment/x-wing -- curl rebel-base
  

  {"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
  

• Set the clustermesh.cilium.io/global annotation on the rebel-base-demo Namespace on mbr-aks-member-1 (cluster 1) to "false" so Services in that Namespace are no longer shared across the cross-cluster network from cluster 1.

  kubectl --context=cluster1 annotate namespace rebel-base-demo clustermesh.cilium.io/global="false" --overwrite
  

  Note

  After changing the Namespace-level clustermesh.cilium.io/global annotation, reconciling each Service within that Namespace is necessary to put the new configuration into effect (e.g. with kubectl --context=cluster1 -n rebel-base-demo apply -f <service-manifest>.yaml). The per-Service service.cilium.io/global annotation, by contrast, takes effect on both clusters within a few seconds.

• Let's validate mbr-aks-member-1 (cluster 1) can reach the rebel-base Service on cluster 1, and mbr-aks-member-2 (cluster 2) can't.

  kubectl --context=cluster1 -n rebel-base-demo exec -ti deployment/x-wing -- curl rebel-base
  

  On cluster 1, we only see local responses because the Namespace is no longer global and remote endpoints aren't imported.

  {"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
  

  Repeat the command on cluster 2.

  kubectl --context=cluster2 -n rebel-base-demo exec -ti deployment/x-wing -- curl rebel-base
  

  On cluster 2, we only see local responses because the Service on cluster 1 is no longer shared.

  {"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
  

Updating a cross-cluster network

The process of adding or removing clusters from a cross-cluster network follows this process.

Delete a cross-cluster network

To delete a cross-cluster network, first remove the members from the cross-cluster network by removing the labels used in the cross-cluster network profile.

Troubleshooting

When members fail to connect, the clustermesh-apiserver LoadBalancer Service on the cluster has no external IP.

Check the clustermesh-apiserver Service on each affected member cluster:

kubectl --context=cluster1 -n kube-system get svc clustermesh-apiserver

If EXTERNAL-IP shows as <pending>, the Service isn't assigned an internal load balancer IP.

NAME                    TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
clustermesh-apiserver   LoadBalancer   10.0.221.162   <pending>     2379:31640/TCP   31s

Inspect the Service events to identify the underlying cause:

kubectl --context=cluster1 -n kube-system describe svc clustermesh-apiserver

A common cause is an AuthorizationFailed (HTTP 403) error, which indicates that the AKS cluster identity doesn't have permission to manage the subnet the cluster connects to.

Events:
  Type     Reason                  Age   From                Message
  ----     ------                  ----  ----                -------
  Normal   EnsuringLoadBalancer    6s    service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  6s    service-controller  Error syncing load balancer: failed to ensure load balancer: ...
                                                             RESPONSE 403: 403 Forbidden
                                                             ERROR CODE: AuthorizationFailed
                                                             {
                                                               "error": {
                                                                 "code": "AuthorizationFailed",
                                                                 "message": "The client '...' with object id '...' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '/subscriptions/.../virtualNetworks/<vnet>/subnets/<subnet>' or the scope is invalid..."
                                                               }
                                                             }

Without this permission, the AKS load balancer controller can't assign an internal IP to the clustermesh-apiserver Service, and the cross-cluster network can't complete the connection.

The AKS cluster identity normally receives a Network Contributor role assignment on the virtual network or subnet when the cluster is created, but in some configurations this assignment isn't created. Manually grant the AKS cluster identity the Network Contributor role on the virtual network for every affected member cluster, then force a reconcile on the cluster using the az aks update command so the new role takes effect.

AKS_IDENTITY=$(az aks show --resource-group ${GROUP} --name ${MEMBER_CLUSTER_1} --query "identity.principalId" -o tsv)
VNET_ID=$(az network vnet show --resource-group ${GROUP} --name <vnet-name> --query id -o tsv)

az role assignment create \
    --assignee-object-id ${AKS_IDENTITY} \
    --assignee-principal-type ServicePrincipal \
    --role "Network Contributor" \
    --scope ${VNET_ID}

az aks update --resource-group ${GROUP} --name ${MEMBER_CLUSTER_1}

Once the cluster is updated with the role assignment, run az fleet clustermeshprofile apply again.

Next steps

• Overview of Fleet Manager multi-cluster networking.
• Fleet Manager Frequently Asked Questions (FAQs).