Share via

Optimized Apache Spark pools cluster to run parallel notebook in Synapse

ADITYA SINGH 0 Reputation points
2025-11-11T13:36:17.43+00:00

I have a cluster(apache spark pool) in azure synapse analytics of the below size
User's image

With the above cluster configuration when I am using this same cluster in my synapse pipeline to run notebooks I am only able to run 8 notebooks in foreach parallelly.

Pipeline level notebook configuration
User's image

Just wanted to understand what can be the best configuration to run more then 8 notebooks parallelly.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Vinodh247 40,031 Reputation points MVP Volunteer Moderator
    2025-11-12T01:31:21.6933333+00:00

    Hi ,

    Thanks for reaching out to Microsoft Q&A.

    increase pool size (more nodes) + raise max concurrent sessions + keep executor footprint small. That combination will allow 12 to 16 parallel notebooks reliably, depending on workload complexity.

    Your current synapse spark pool (Large nodes, 3to 6 autoscale, 16 vCores / 128 GB each) and notebook settings (Small executors, 1 to 2 dynamically allocated) limit concurrency. To run more than 8 notebooks in parallel, optimise as follows:

    1. Increase node count > move autoscale upper limit to 8–10 nodes.
    2. Use smaller executors > keep executor size Small (4 vCores, 28 GB) to maximise parallel sessions.
    3. Lower per notebook resource use > limit each notebook to 1 executor, dynamic allocation enabled.
    4. Adjust concurrency setting > in synapse Spark pool configuration, raise “Maximum concurrent sessions” (default is 8).
    5. Avoid cache contention > reduce Intelligent cache below 50 % if memory pressure occurs.

    Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.