Flux in a Hub $ Spoke Architecture - How to access remote AKS Cluster with Azure RBAC enabled and local_user access disabled

Question

Flux in a Hub $ Spoke Architecture - How to access remote AKS Cluster with Azure RBAC enabled and local_user access disabled

Hering, Martin 0

Hi there,

I'm currently setting up a hub-and-spoke GitOps architecture using the AKS GitOps (Flux) extension. Flux is installed in a central (hub) AKS cluster and is intended to manage application and infrastructure deployments across multiple spoke AKS clusters.

All spoke clusters are configured with Azure RBAC enabled and local_user access disabled, as per our security requirements. This setup prevents the use of static kubeconfigs tied to local cluster admin credentials.

Flux uses a kubeConfig.secretRef to authenticate to the spoke clusters. However, since local_user is disabled and the only supported login mechanism in this scenario is via Azure AD (kubelogin), i am wondering how Flux is able to authenticate to those spoke clusters. We do not want to re-enable local_user as it would introduce a security backdoor by bypassing RBAC.

Is there a Microsoft-supported way to enable Flux to authenticate to remote AKS clusters using Azure AD (i.e., without local_user)?

Can Flux be configured to use a workload identity, managed identity, or service principal to authenticate to remote clusters in this kind of setup?

Is this multi-cluster hub-to-spoke model fully supported under these security constraints, or is there a recommended alternative?

Any help would be highly appreciated

Best regards,

Martin

1 answer

Your answer

Answer 1

Suwarna S Kale 1,906

Hello Hering, Martin

Thank you for posting your question in the Microsoft Q&A forum.

To enable Flux to authenticate to Azure RBAC-enabled spoke clusters without local_user, Microsoft recommends using Azure Workload Identity a fully supported solution that aligns with zero-trust principles. By configuring Flux with a managed identity federated to Azure AD, the hub cluster can securely access spokes via short-lived tokens, eliminating static credentials. This approach requires annotating Flux’s service account with the managed identity’s client ID and granting it Azure Kubernetes Service Cluster User Role on spoke clusters. Alternatively, Cluster API (CAPZ) or Azure Arc can orchestrate multi-cluster management while maintaining Azure AD authentication. Both methods ensure compliance with security constraints, as they bypass local_user entirely. For production environments, this architecture is Microsoft-validated, provided spoke clusters run Kubernetes v1.22+.

If the above answer helped, please do not forget to "Accept Answer" as this may help other community members to refer the info if facing a similar issue. Your contribution to the Microsoft Q&A community is highly appreciated.

Akshay kumar Mandha 2,690 Reputation points Microsoft External Staff

2025-04-21T09:33:26.76+00:00

Hi Hering, Martin,
Based on your query,
For Flux in a Hub $ Spoke Architecture - How to access remote AKS Cluster with Azure RBAC enabled and local user access disabled Suwarna S Kale mentioned in answer section please review it once for more information please refer to the documentation Azure Workload Identity and GitOps for AKS.
Please let me know if you have any further query
Hering, Martin 0 Reputation points

2025-04-21T18:19:38.8033333+00:00
Hi Suwarna,

Thanks for your quick response!

I’ve been thinking about using workload identities as well. To make that work, I would assign both the kustomize-controller and helm-controller a federated credential, pointing to the workload identity I’ve already set up. I’m currently using that same identity to allow Flux in the hub cluster to access our Azure DevOps repository.

I considered using the Azure Kubernetes Service RBAC Admin role because i need to deploy applications within those spoke clustes, i think assigning the Azure Kubernetes Service Cluster User Role might not be enough.

That said, I think the remaining challenge is authentication via kubeConfig.secretRef. Since Azure RBAC is enabled and local_user access is disabled on the spoke clusters, I need to run:

bash kubelogin convert-kubeconfig -l workloadidentity

...to convert the kubeconfig to use workload identity-based auth before injecting it into Flux as a secret.

My main question is: how does the flux controller know about kubelogin? Unfortunately kubelogin is not installed within the controllers. I did exec into the running containers and could not find kubelogin as a binary.

Am i missing something here ?

Best regards,

Martin
Hering, Martin 0 Reputation points

2025-04-21T20:44:40.0733333+00:00
Hi Suwarna,

I’ve attempted the setup using Azure Workload Identity, but unfortunately it failed with an authorization error. I was able to make it work by enabling local_user and not using workload identity, but that’s not the outcome I want, as it bypasses Azure RBAC.

Here’s what I did during my test:

I used my existing workload identity and created federated credentials for both the kustomize-controller and helm-controller.

I loaded the kubeconfig from the spoke cluster using:

az aks get-credentials --resource-group <rg-name> --name <cluster-name> --overwrite-existing export KUBECONFIG=/path/to/kubeconfig kubelogin convert-kubeconfig -l workloadidentity

Then I deployed this kubeconfig as a secret into the hub cluster, to be referenced by Flux via kubeConfig.secretRef.

I assigned the workload identity the Azure Kubernetes Service RBAC Cluster Admin role on the spoke cluster to eliminate any permission-related issues.

However, reconciliation still failed with the following error:

{ "lastTransitionTime": "2025-04-21T20:06:53+00:00", "message": "Namespace/traefik dry-run failed (Unauthorized): failed to get API group resources: unable to retrieve the complete list of server APIs: v1: Unauthorized", "reason": "ReconciliationFailed", "status": "False", "type": "Ready" }

Since the role assignment is correct, I believe the issue is due to failed authentication via kubelogin, possibly because the kubelogin binary is not available inside the Flux controller pods. I’ve also verified that the required annotations and labels for workload identity are present on both the service accounts and controllers.

Thank you very much for assistance

Martin
Akshay kumar Mandha 2,690 Reputation points Microsoft External Staff

2025-04-22T14:14:10.2666667+00:00

Hi Hering, Martin,
Thank you for taking the time to share the details with us that you have triedBased on the error you've reported and considering that the annotations and labels have been verified and are in place it appears that the issue may be related to Kubelogin. Could you please check Kubelogin might need to be installed or may not be available within the Flux controller pod please verify it once

If you have any further questions or need additional assistance, please feel free to reach out. We're here and happy to help
Akshay kumar Mandha 2,690 Reputation points Microsoft External Staff

2025-04-23T09:00:01.71+00:00

Hi Hering, Martin,
Just checking to see on previous comment if you have any other query, please let me know
Thank you
Hering, Martin 0 Reputation points

2025-04-23T18:58:58.0866667+00:00
Hi @Akshay kumar Mandha ,

I noticed that only the source-controller and kustomize-controller have the expected labels and annotations related to workload identity in place. However, the helm-controller does not, despite being deployed in the same way. I set up all controllers using Terraform, and the setup process was identical across them.

Here are the steps I’ve already taken to troubleshoot:

Verified that all federated identity credentials are correctly configured in the UI for the user-assigned identity.

Double-checked the azurerm_federated_identity_credential resources for all three controllers (source-controller, kustomize-controller, and helm-controller) and confirmed they are being created using a loop over the same list.

Inspected the deployed annotations and labels on the controllers — only helm-controller is missing them.

Confirmed that kubelogin is not installed on all controllers (which might be related).

Here is a snippet of my Terraform configuration:

flux_service_accounts = ["source-controller", "helm-controller", "kustomize-controller"] resource "azurerm_federated_identity_credential" "fluxControllerFederation" { for_each = toset(var.flux_service_accounts) name = "flux_${each.key}_k8s" ... subject = "system:serviceaccount:flux-system:${each.value}" }

and the Flux extension configuration:

resource "azapi_resource" "fluxExtensions" { ... configurationSettings = { "workloadIdentity.enable" = "true" "workloadIdentity.azureClientId" = azurerm_user_assigned_identity.fluxSourceControllerIdentity.client_id } }
Akshay kumar Mandha 2,690 Reputation points Microsoft External Staff

2025-04-24T07:30:50.9933333+00:00
Hi Hering, Martin
Based on you latest comment missing helm controller missing. you can inspect the helm-controller service account configuration using the following command:
kubectl get serviceaccount helm-controller -n flux-system -o yaml

Please refer the below Kubernetes document for command line

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server

Confirm that the Azure client ID is correctly linked for helm-controller
Check the Azure Client ID from Flux Extension:

Run the following command to retrieve the Azure Client ID configured for Flux

az k8s-extension show \ --resource-group <YOUR_RESOURCE_GROUP> \ --cluster-name <YOUR_CLUSTER_NAME> \ --name flux

Look for the workloadIdentity.azureClientId value in the output. This is the client ID used by Flux
Verify the Federated Identity Credential:

Run the following command to list the federated identity credentials associated with the user-assigned identity

az identity federated-credential list \ --identity-name <YOUR_USER_ASSIGNED_IDENTITY_NAME> \ --resource-group <YOUR_RESOURCE_GROUP>

Please refer the below commands line documents
az connectedk8s show
az identity federated-credential

To set up kubelogin, first install the Azure Kubernetes CLI by running the command az aks install-cli. Once that’s done, you’ll use the command kubelogin convert-kubeconfig to update your kubeconfig file so it can handle authentication methods like workload identity. You’ll need to provide some specific information, like your Microsoft Entra tenant ID (--tenant-id), the app’s client ID (--client-id), and the server ID (--server-id).
Please refer the below document for more information
How authentication works
If you have any further query, please let me know
Hering, Martin 0 Reputation points

2025-04-24T08:49:31.2133333+00:00
Hi Akshay kumar Mandha,

Thanks for your quick response.

I have a quick follow-up question regarding the Helm controller:

Is it really necessary to manually configure the service account for the Helm controller?

Is there no way to automate this setup, similar to how it works with the Source and Kustomize controllers?

Why does it work out of the box for Source and Kustomize controllers, but not for the Helm controller,m when setting this up via terraform?

I'll try out the approach you suggested and will get back to you once I've tested it.

Thanks again and best regards,

Martin

Share via

Flux in a Hub $ Spoke Architecture - How to access remote AKS Cluster with Azure RBAC enabled and local_user access disabled

1 answer

Your answer