Unable to Start Apache Spark Pool. Fails with SessionCreationTimeoutException: Session creation timed out.

Question

Unable to Start Apache Spark Pool. Fails with SessionCreationTimeoutException: Session creation timed out.

MikelleRogers-6194 20

When I run a pipeline, it fails with ]Cluster was in terminal state=[Cancelled] before it reached 'Ready' state.

When I run a new notebook, it fails with SessionCreationTimeoutException: Session creation timed out.

I am on Spark version 3.5

SAI JAGADEESH KUDIPUDI 3,640 Reputation points Microsoft External Staff Moderator

2026-06-23T23:17:58.6566667+00:00
Hi @MikelleRogers-6194 ,
the Spark cluster can't reach the Ready state in time, so the Livy session never gets created. This is a cluster startup/provisioning issue, not a code or job-execution problem.

Scale the pool up and back down (or restart it) to force a fresh cluster allocation — this clears most transient startup failures.

Retry with a smaller node size and minimum executors — if it starts, the issue is capacity/SKU-related.

Remove and re-attach custom packages (common after a 3.4 → 3.5 upgrade) and check the library install logs for conflicts.

Enable Livy error classification for a precise cause: set livy.rsc.synapse.error-classification.enabled = true.

Reference: Handle Livy errors on Apache Spark in Synapse [learn.microsoft.com]

If it still fails at a small size with no packages, please share your requested details in private message
so we can check the backend cluster allocation.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.
MikelleRogers-6194 20 Reputation points

2026-06-24T01:14:22.49+00:00

Where do I enable livy error classification?
Ravi Kiran Pagidi 90 Reputation points

2026-06-25T19:20:05.68+00:00
Hi @MikelleRogers-6194 ,

You enable it as a Spark configuration.

In Synapse Studio:

Go to Manage.

Select Apache Spark configurations.

Create a new configuration, or edit the configuration attached to your Spark pool.

Add this property:

livy.rsc.synapse.error-classification.enabled true

Publish the changes.

Attach that Spark configuration to the Spark pool, notebook session, or Spark job definition.

Stop any existing Spark session and start a new one.

After the next failure, check Monitor > Apache Spark applications. The error details should show a more specific classified Livy/Spark error instead of only the generic timeout/dead session message.

Answer accepted by question author

SAI JAGADEESH KUDIPUDI 3,640 Microsoft External Staff Moderator

Hi @MikelleRogers-6194 ,

Thanks for reaching out Microsoft Q&A

Based on the symptoms shared (SessionCreationTimeoutException: Session creation timed out, CLUSTER_CREATION_TIMED_OUT, and Spark pools failing to reach the Ready state), the issue was related to the workspace encryption configuration.
Root Cause

Our investigation found that Spark session creation was failing before Livy and YARN services could start. The backend cluster could not be provisioned because the Customer Managed Key (CMK) referenced by the Synapse workspace had expired.

As a result:

All Spark pools in the workspace failed during cluster creation.
Errors such as SessionCreationTimeoutException and CLUSTER_CREATION_TIMED_OUT were observed.
No Spark Application ID or Livy logs were generated because cluster provisioning never completed.
The issue affected multiple Spark pools across Spark versions 3.4 and 3.5, including newly created pools.

The backend logs showed a KeyVaultAccessForbidden error indicating that the encryption key being used by the workspace had expired.

Resolution

The encryption key in Azure Key Vault was renewed/rotated and the workspace encryption reference was validated. After the key was renewed, Spark cluster provisioning succeeded and notebook sessions started successfully.

You can continue using the existing Key Vault and key by renewing it. If renewing the existing key does not resolve the issue, create a new key version and update the workspace to reference it.

After renewing the encryption key, the Spark pools started successfully and the issue was resolved.

We're glad to hear that renewing the key worked and that everything is now functioning correctly. Thank you for confirming the resolution.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

0 comments

1 additional answer

Your answer

SAI JAGADEESH KUDIPUDI 3,640 Reputation points Microsoft External Staff Moderator

2026-06-23T23:17:58.6566667+00:00

Hi @MikelleRogers-6194 ,
the Spark cluster can't reach the Ready state in time, so the Livy session never gets created. This is a cluster startup/provisioning issue, not a code or job-execution problem.

Scale the pool up and back down (or restart it) to force a fresh cluster allocation — this clears most transient startup failures.

Retry with a smaller node size and minimum executors — if it starts, the issue is capacity/SKU-related.

Remove and re-attach custom packages (common after a 3.4 → 3.5 upgrade) and check the library install logs for conflicts.

Enable Livy error classification for a precise cause: set livy.rsc.synapse.error-classification.enabled = true.

Reference: Handle Livy errors on Apache Spark in Synapse [learn.microsoft.com]

If it still fails at a small size with no packages, please share your requested details in private message
so we can check the backend cluster allocation.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.
MikelleRogers-6194 20 Reputation points

2026-06-24T01:14:22.49+00:00

Where do I enable livy error classification?
Ravi Kiran Pagidi 90 Reputation points

2026-06-25T19:20:05.68+00:00

Hi @MikelleRogers-6194 ,

You enable it as a Spark configuration.

In Synapse Studio:

Go to Manage.

Select Apache Spark configurations.

Create a new configuration, or edit the configuration attached to your Spark pool.

Add this property:

livy.rsc.synapse.error-classification.enabled true

Publish the changes.

Attach that Spark configuration to the Spark pool, notebook session, or Spark job definition.

Stop any existing Spark session and start a new one.

After the next failure, check Monitor > Apache Spark applications. The error details should show a more specific classified Livy/Spark error instead of only the generic timeout/dead session message.

Answer 1

Hi Mikelle,

Since both the pipeline and a new notebook fail before the Spark pool reaches the Ready state, this is likely a Spark pool startup/session creation issue rather than an issue with your notebook code.

Please check the below items:

Go to Synapse Studio > Monitor > Apache Spark applications and open the failed session. Check whether there is a more specific error code behind the timeout.

Check whether the workspace has enough available Spark vCore quota. If the pool cannot allocate the requested nodes, the session can fail before startup.

Try creating a small test Spark pool with minimum nodes and no custom packages/configuration. If that starts, compare it with the failing pool.

If the pool has custom libraries, packages, or Spark configuration attached, temporarily remove them and test again.

Check Azure Service Health and Synapse known issues for the workspace region.

If this started suddenly and the same pool worked earlier, open a Microsoft support request with the workspace name, Spark pool name, region, UTC timestamp, failed Livy/session ID, and the exact error message.

Docs:

Interpret Synapse Spark/Livy errors: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-handle-livy-error

Synapse Spark pool configurations: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-pool-configurations

Synapse Spark library troubleshooting: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-troubleshoot-library-errors

Synapse Spark 3.5 runtime: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-35-runtime

Spark 3.5 is a supported Synapse runtime, so I would first isolate quota/capacity, custom packages, and pool configuration. If even a small clean pool fails in the same workspace/region, it is likely something Microsoft support needs to check on the backend.