Hi Microsoft team,
I am testing an Azure Marketplace Managed Application that deploys an Azure Machine Learning workspace, ACR, storage account, Key Vault, user-assigned managed identity, and AML compute cluster inside the managed resource group.
I am trying to restrict customer access using the Marketplace Managed Application Customer Access = Restricted option, while still allowing the customer to submit Azure ML training jobs.
Scenario
The Managed Application deploys:
AML compute cluster
Azure Container Registry
Storage account
Key Vault
User-assigned managed identity attached to the AML compute
The UAMI has the following role assignments:
Reader on the storage account
Storage Account Contributor on the storage account
Storage Blob Data Contributor on the storage account
Contributor on the Azure ML workspace
Reader on Key Vault
Key Vault Secrets User on Key Vault
Inside the Azure ML training container, I run Python code that uses the UAMI to call:
from azure.identity import ManagedIdentityCredential
from azure.mgmt.storage import StorageManagementClient
credential = ManagedIdentityCredential(client_id=UAMI_CLIENT_ID)
client = StorageManagementClient(
credential,
subscription_id=SUBSCRIPTION_ID,
)
keys = client.storage_accounts.list_keys(
resource_group_name=RESOURCE_GROUP,
account_name=STORAGE_ACCOUNT_NAME,
)
The purpose of this test is to confirm whether the UAMI can access the storage account keys and then write outputs/logs to Blob Storage.
Important finding
When I deploy the Managed Application with Customer Access = Full, the Azure ML training job works successfully. The job can write the output files and logs to the storage account.
However, when I deploy the same Managed Application with Customer Access = Restricted, the same training job fails on list_keys, even though the UAMI has Storage Account Contributor and the generated deny assignment appears to allow storage account actions.
For testing, I made the allowed actions broad:
Allowed control actions:
Microsoft.MachineLearningServices/workspaces/*;Microsoft.Storage/storageAccounts/*
Allowed data actions:
Microsoft.Storage/storageAccounts/*
Actual deny assignment JSON
I checked the actual deny assignment created on the managed resource group using:
az rest \
--method get \
--url "https://management.azure.com${RG_ID}/providers/Microsoft.Authorization/denyAssignments?api-version=2022-04-01" \
-o jsonc
The deny assignment contains:
{
"permissions": [
{
"actions": [
"*"
],
"dataActions": [
"*"
],
"notActions": [
"*/read",
"Microsoft.MachineLearningServices/workspaces/*",
"Microsoft.Storage/storageAccounts/*"
],
"notDataActions": [
"*/read",
"Microsoft.Storage/storageAccounts/*"
]
}
],
"principals": [
{
"id": "00000000-0000-0000-0000-000000000000",
"type": "SystemDefined"
}
],
"condition": "@Subject[ResourceId] StringNotStartsWithIgnoreCase '/subscriptions/<subscription-id>/resourceGroups/<managed-resource-group>' && @Subject[tid] StringNotEqualsAnyOfIgnoreCase {'<publisher-tenant-id>'}",
"scope": "/subscriptions/<subscription-id>/resourceGroups/<managed-resource-group>"
}
So the actual deny assignment does include:
Microsoft.Storage/storageAccounts/*
under notActions.
Error
Despite that, the training job fails with:
azure.core.exceptions.HttpResponseError: (AuthorizationFailed)
The client '<UAMI_CLIENT_ID>' with object id '<UAMI_PRINCIPAL_ID>' does not have authorization to perform action
'Microsoft.Storage/storageAccounts/listKeys/action'
over scope '/subscriptions/<subscription-id>/resourceGroups/<managed-resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account-name>'
or the scope is invalid. If access was recently granted, please refresh your credentials.
Code: AuthorizationFailed
Message: The client '<UAMI_CLIENT_ID>' with object id '<UAMI_PRINCIPAL_ID>' does not have authorization to perform action
'Microsoft.Storage/storageAccounts/listKeys/action'
Why this is confusing
I expected this to work because:
The UAMI has Storage Account Contributor on the exact storage account.
The action being denied is:
Microsoft.Storage/storageAccounts/listKeys/action
The deny assignment has this exception in notActions:
Microsoft.Storage/storageAccounts/*
The exact same training job works when the Managed Application is deployed with Customer Access = Full.
The issue only appears when the Managed Application uses restricted customer access.
Questions
Is this expected behavior for Azure Marketplace Managed Applications with restricted customer access?
Should Microsoft.Storage/storageAccounts/* in notActions allow:
Microsoft.Storage/storageAccounts/listKeys/action
Or does listKeys/action need to be explicitly added, even when the wildcard is present?
For example, should the allowed control actions include:
Microsoft.Storage/storageAccounts/listKeys/action
separately?
Also, does the Managed Application deny assignment apply to user-assigned managed identities created inside the managed resource group, even if those identities have explicit Azure RBAC roles on the storage account?
Finally, for Azure ML training jobs that need to write models/logs to Storage using managed identity, what are the recommended allowed control actions and allowed data actions for a restricted-customer-access Managed Application?
The main point I am trying to understand is:
Why does the UAMI fail Microsoft.Storage/storageAccounts/listKeys/action only under restricted customer access, even though the actual deny assignment includes Microsoft.Storage/storageAccounts/* in notActions, and the same deployment works with Customer Access = Full?Title: Azure Managed Application restricted customer access blocks UAMI listKeys despite Microsoft.Storage/storageAccounts/* in deny assignment notActions