Pipeline to wait until spark cluster is available

@Kumar, Arun

You're correct, when multiple Spark notebooks are triggered at the same time in a small Spark pool, they may fail with the 6002 - timed out while waiting for cluster error if there aren't enough available nodes to serve all sessions concurrently.

Azure Synapse Spark pools don’t currently support built-in queuing for notebook executions. However, here are a few options to help you manage this situation:

Workaround Options:

Add Retry Logic with Until Activity (ADF) - You can build a custom loop using the Until activity in your pipeline to wait until the cluster is ready. For example:

Try to run a lightweight notebook or status check script.

If it fails due to unavailability, wait (e.g., 1-2 minutes), then retry.

Break out of the loop once the notebook runs successfully or the cluster is ready.

Stagger Notebook Executions with Wait Activity - If you're launching multiple notebooks, consider spacing them out using Wait activities. This won't queue them dynamically, but it helps avoid overloading the pool at once.

Use Azure Monitor + Logic Apps - You can set up alerts on your Spark pool (e.g., when CPU or session count is low) and trigger Logic Apps or pipelines based on availability. This creates an automated, resource-aware trigger system.

Consider Scaling the Spark Pool - If possible, review your Spark pool size or consider enabling dynamic allocation (if applicable) to automatically add resources when demand increases.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.