Unable to generate MSI token from Synapse while writing data to ServiceBus and EventHub

Hi ,

Thanks for reaching out to Microsoft Q&A.

yes, this is a known limitation with Synapse Spark and MsSparkUtils.

The MsSparkUtils.credentials.getToken() method in Synapse is limited to a predefined list of Azure services (like Azure SQL, Storage, etc.). Currently, Azure Service Bus and Event Hub are not included in that list of supported audience values.

Docs Reference The audience parameter must match one of the supported services (e.g., https://storage.azure.com/, https://database.windows.net/).

Workaround Options:

1. Use a Custom Token Provider with MI (Outside MsSparkUtils)

If you're running in a Synapse Spark pool with Managed Identity enabled, you can still obtain a token using the Azure Identity SDK via a custom JAR:

import com.azure.identity.DefaultAzureCredentialBuilder
import com.azure.core.credential.TokenRequestContext
val credential = new DefaultAzureCredentialBuilder().build()
val requestContext = new TokenRequestContext().addScopes("https://servicebus.azure.net/.default")
val accessToken = credential.getToken(requestContext).block()
val token = accessToken.getToken

• This gives you a valid token to use with Service Bus or Event Hub.
• You’ll need to bring in Maven dependencies like:
  • com.azure:azure-identity
  • com.azure:azure-core

2. Use Connection String (Not Ideal but Works)

If MSI isn’t feasible and you’re blocked, fallback to a connection string in a keyvault backed Spark config (yes, not ideal security-wise, but a practical workaround).

To sum up:

1. You cannot get MSI tokens for ServiceBus/EventHub using MsSparkUtils.credentials.getToken().
2. Instead, use the Azure Identity SDK with DefaultAzureCredential to manually fetch the token in Spark.
3. Or, temporarily use a connection string from keyvault if MSI is blocked.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.

Please 'Accept as Answer' and click the 'Upvote'(Thumbs-up) button if the reply was helpful. This will help us to close this thread.

Ravi Shankar Gupta Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

The answer provided is not working . I tried using the approach but getting this error.

2025-03-26 10:37:55,516 INFO ChainedTokenCredential [Driver]: Azure Identity => Attempted credential EnvironmentCredential is unavailable.
2025-03-26 10:37:56,594 INFO ChainedTokenCredential [ForkJoinPool.commonPool-worker-3]: Azure Identity => Attempted credential ManagedIdentityCredential is unavailable.
2025-03-26 10:37:56,638 INFO ChainedTokenCredential [ForkJoinPool.commonPool-worker-3]: Azure Identity => Attempted credential SharedTokenCacheCredential is unavailable.
2025-03-26 10:37:56,643 INFO ChainedTokenCredential [ForkJoinPool.commonPool-worker-3]: Azure Identity => Attempted credential IntelliJCredential is unavailable.
2025-03-26 10:37:56,735 INFO ChainedTokenCredential [ForkJoinPool.commonPool-worker-3]: Azure Identity => Attempted credential AzureCliCredential is unavailable.
2025-03-26 10:37:56,765 INFO ChainedTokenCredential [ForkJoinPool.commonPool-worker-5]: Azure Identity => Attempted credential AzurePowerShellCredential is unavailable.

There is no issue with MSI as the same setup is working perfectly fine with HDInsight cluster.

Sample code used:

TokenCredential credential = new DefaultAzureCredentialBuilder()
        .managedIdentityClientId(MSI_CLIENT_ID).build();
//https://servicebus.azure.net/.default
TokenRequestContext requestContext = new TokenRequestContext().addScopes("https://servicebus.azure.net/.default");
AccessToken accessToken = credential.getToken(requestContext).block();