Trouble connecting from synapse notebook to eventhub using managed identity

I ran into this before when trying to use Synapse notebooks with managed identity, and the issue wasn’t the code. The Spark session simply wasn’t actually running under the managed identity, even though “Run as managed identity” was enabled.

In Synapse, the managed identity only works if the Spark pool itself has the right role on the Event Hub. The notebook session inherits from the pool, not the workspace.

Make sure the Synapse workspace managed identity or the Spark pool managed identity has:

Azure Event Hubs Data Sender on the Event Hub namespace.

Without that, the IMDS call from the cluster will fail and you’ll get that exact “Connection to IMDS endpoint cannot be established” error.

Once I added the role directly on the Event Hub namespace and restarted the Spark session, the token request worked immediately.

Thank you Tharangini Anandan for posting your question on Microsoft Q&A and for sharing the exact error you’re seeing.

The error

com.azure.identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. Connection to IMDS endpoint cannot be established, Connect timed out

typically indicates that the runtime environment is unable to reach the Azure Instance Metadata Service (IMDS) endpoint (169.254.169.254), which managed identity credentials rely on. This is common in certain Synapse notebook execution contexts where the underlying compute doesn’t fully support IMDS access, especially when using libraries like the Azure SDK for Java/Scala directly in a PySpark notebook session.

While mssparkutils works for Key Vault because it uses a different internal authentication path optimized for Synapse, Event Hubs connectivity via managed identity often requires a different approach. One reliable workaround is to use Azure Active Directory service principal authentication (with a secret or certificate) temporarily if MI isn’t accessible from your notebook runtime, or alternatively, pull the Event Hubs connection string via mssparkutils from Key Vault and use it directly with the Event Hubs Kafka or Spark connector, avoiding the need for ManagedIdentityCredential in the notebook code itself.

Also, confirm whether you’re running the notebook on Spark pool (not serverless), managed identity via IMDS is only consistently available in dedicated Spark sessions, not in serverless or certain lightweight runtimes.

Have you tried this connection from a dedicated Spark pool rather than a serverless notebook session? And are you using PySpark’s built-in Event Hubs connector (via eventhubs format) or the Azure SDK directly in a Scala/Java cell? That distinction can affect whether IMDS is reachable at all.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Thanks

Pratyush