Guidance on Managed Identity Approach for ADLS Gen2

From a Unity Catalog and ADLS Gen2 perspective, the recommended pattern is to design storage credentials and managed identities around storage accounts or data domains, not individual containers, and to use Azure RBAC for least-privilege access.

1. Recommended approach from a security and governance perspective

For Azure Databricks + Unity Catalog on Azure, the guidance is:

• Use separate Azure Databricks Access Connectors (managed identities) for different storage accounts or data domains.
• Apply least-privilege permissions using Azure RBAC roles on the storage account.

In a scenario where multiple domains/teams share a single storage account but are isolated by containers, two main options align with this guidance:

• Single managed identity per storage account (or per broad data domain) and use container-level RBAC (for example, Storage Blob Data Contributor scoped to specific containers) combined with Unity Catalog external locations and grants.
• Multiple managed identities only when there is a clear need to separate data domains at the identity level (for example, strong blast-radius isolation, different operational ownership, or different trust boundaries) even within the same storage account.

Unity Catalog’s primary isolation and governance mechanism is at the catalog/schema/table and external location level, with storage credentials (backed by access connectors) providing the authentication to storage. The documentation explicitly recommends separate access connectors for different storage accounts or data domains, not necessarily per container.

2. Best practice for enterprise-scale Databricks + ADLS Gen2 + Unity Catalog

Enterprise guidance for Azure authentication and storage credentials in Unity Catalog includes:

• Use Azure Databricks Access Connectors (managed identities) as storage credentials.
• Create one storage credential per environment (dev, staging, prod) and per business unit when data segregation is required.
• Use separate Access Connectors for different storage accounts or data domains.
• Apply least-privilege permissions using Azure RBAC.

Applied to a single storage account with multiple domain-specific containers:

• Treat each data domain as a logical boundary. If domains map to distinct business units or trust boundaries, create separate access connectors (and thus separate managed identities) and storage credentials per domain, even if they share the same storage account.
• If domains are more organizational than security boundaries (for example, multiple teams within the same business unit), a single access connector/managed identity per environment and storage account, with container-level RBAC and Unity Catalog grants, is typically sufficient and simpler.

3. RBAC, operational, or maintenance concerns with multiple managed identities in the same storage account

Using multiple managed identities (via multiple access connectors) against the same storage account introduces:

• RBAC complexity:
  • Each managed identity must be granted appropriate roles (for example, Storage Blob Data Contributor) at the storage account or container scope.
  • More identities mean more role assignments to manage and audit.
• Operational overhead:
  • Additional storage credentials in Unity Catalog to create and maintain.
  • More external locations referencing different storage credentials.
• Governance benefits:
  • Clear separation of responsibilities and blast radius: if one identity’s permissions are misconfigured, it affects only the containers assigned to that identity.
  • Easier per-domain monitoring and revocation by revoking RBAC for a specific identity.

The managed identity best-practice guidance also notes that user-assigned identities can reduce administration by centralizing role assignments, but multiple identities are supported and can be used for granular permissions when needed.

4. Single managed identity with container-level RBAC sufficiency and security

A single managed identity (via one access connector) with container-level RBAC can be secure and aligned with best practices when:

• The storage account and its containers belong to the same broad data domain or business unit.
• Unity Catalog is used to enforce domain-level isolation via:
  • Separate external locations per container.
  • Separate catalogs/schemas per domain.
  • Catalog grants controlling which principals can use which external locations.
• Azure RBAC is scoped to give the managed identity only the necessary access (for example, Storage Blob Data Contributor at container scope rather than entire account if tighter control is required).

Unity Catalog’s design explicitly separates when storage credentials are required (external locations, external tables, volumes) from managed tables, and recommends using managed tables and Unity Catalog for access control wherever possible. For external storage, the combination of a single managed identity, container-scoped RBAC, and strict Unity Catalog grants is a valid and scalable pattern.

Summary design guidance for the described scenario:

• Default pattern:
  • One access connector/managed identity per environment and storage account (or per broad data domain).
  • Use container-level RBAC and Unity Catalog external locations and grants for domain isolation.
• Use multiple managed identities within the same storage account when:
  • Domains map to distinct business units or trust boundaries.
  • Separate operational ownership or compliance requirements justify identity-level separation.
• Always:
  • Apply least-privilege RBAC.
  • Use separate storage credentials per environment and per business unit when data segregation is required.
  • Avoid over-provisioning a single identity with broad access beyond what Unity Catalog needs for its external locations.

References:

• Phase 3: Design Unity Catalog architecture
• Phase 5: Design storage architecture
• Connect to an Azure Data Lake Storage Gen2 (ADLS Gen2) external location
• Create a storage credential that accesses ADLS Gen2
• Use Azure managed identities in Unity Catalog to access storage
• Managed identity best practice recommendations