Generic error when running a Synapse pipeline containing a dataflow or trying to start a debug session

Question

Generic error when running a Synapse pipeline containing a dataflow or trying to start a debug session

MS 0

I have 2 Synapse instances in UK South. Error only occurs in one of them and doesn't seem to show up in the workspace logs - 'Internal server error executing the request.' This happens when trying to start debug cluster or run a pipeline with a dataflow inside it.

Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-10-21T02:41:52.6666667+00:00

Hi MS,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

Thanks,
Manoj
MS 0 Reputation points

2025-10-21T13:41:07.0066667+00:00

Hi Manoj, I agree with you it may be network related as its the workspace using the Managed Virtual Network and Private endpoints where the debug cluster is failing to start. However I can't see a means of configuring outbound traffic from Synapse with this setup? I've checked that there are no firewalls or nsg's impacting traffic.

Kind regards, Mark
Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-10-21T17:15:45.8866667+00:00

Hi MS,

You’re right when a Synapse workspace uses a Managed VNet with private endpoints, you don’t get a way to manually set or control outbound traffic. Synapse handles all outbound communication on its own using Microsoft-managed connections.

In this setup, Synapse needs to reach a few key services (like your storage account, Azure Data Lake, and the Synapse control plane) to start the debug cluster. If it can’t reach any of these, the cluster start will fail and show that same “Internal server error” message.

Few quick things you can check:

1.Make sure all your private endpoints are in Succeeded state and approved especially for your workspace storage and any linked resources like Key Vault or SQL.

2.If your storage account has firewall rules, make sure the Synapse managed outbound IPs are allowed in that list. You can find these IPs in your workspace under Networking or Managed private endpoints.

3.Also check that public network access isn’t fully turned off for required services unless all private endpoints are properly set up.

If you need to control outbound network rules yourself (like adding NSGs or custom routes), you’d have to use a customer-managed VNet instead of a managed one that’s the only option where outbound traffic can be customized.

Please check those points and let me know how it goes. I’ll be happy to help you go through the private endpoint or firewall settings if it’s still not working.

Thanks,
Manoj
Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-10-23T08:54:45.5033333+00:00

Hi MS,

We haven't heard back from you. Please reply if you have any questions in this matter and we will gladly continue the discussion.

Thanks,
Manoj
Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-10-24T08:53:57.5633333+00:00

Hi MS,

I hope the steps provided above were helpful. Please let me know if there's anything else I can assist you with.

Thanks,
Manoj
MS 0 Reputation points

2025-10-27T17:12:47.0233333+00:00
Hi Manoj,

I've checked all Managed Private Endpoints in Synapse are in Succeeded State

All storage accounts have firewall rules but are connected to Synapse via private endpoint

as above

Kind regards, Mark
Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-10-28T10:50:25.73+00:00

Hi MS,

Based on the error you are seeing it looks like the problem might be related to how your Synapse workspace is set up, especially with the network configuration.

Here are some steps to help you fix the issue:

1.Make sure all your private endpoints (like for storage accounts or key vault) are properly set up and marked as "Succeeded." These need to be working correctly for Synapse to connect to them. Also, check the firewall settings on these services. Make sure that Synapse’s IP addresses are allowed in your storage account’s firewall.

2.Your Managed Virtual Network should be able to reach essential services like your storage and the Synapse control plane. Double-check that everything is connected properly. If you are using private endpoints, make sure that services can still communicate, even if public access is turned off for some of them.

3.Check that your Integration Runtime is working properly in Synapse Studio. If needed, try restarting or creating a new debug cluster to ensure it runs smoothly.

4.Make sure you haven’t reached any resource limits in your Synapse workspace. You can go to the "Monitor" section to check if any resources are being overused or if there are limitations.

Do let us know if you have any questions and concerns.

Thanks,
Manoj

1 answer

Your answer

Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-10-21T02:41:52.6666667+00:00

Hi MS,
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

Thanks,
Manoj
MS 0 Reputation points

2025-10-21T13:41:07.0066667+00:00

Hi Manoj, I agree with you it may be network related as its the workspace using the Managed Virtual Network and Private endpoints where the debug cluster is failing to start. However I can't see a means of configuring outbound traffic from Synapse with this setup? I've checked that there are no firewalls or nsg's impacting traffic.

Kind regards, Mark
Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-10-21T17:15:45.8866667+00:00

Hi MS,

You’re right when a Synapse workspace uses a Managed VNet with private endpoints, you don’t get a way to manually set or control outbound traffic. Synapse handles all outbound communication on its own using Microsoft-managed connections.

In this setup, Synapse needs to reach a few key services (like your storage account, Azure Data Lake, and the Synapse control plane) to start the debug cluster. If it can’t reach any of these, the cluster start will fail and show that same “Internal server error” message.

Few quick things you can check:

1.Make sure all your private endpoints are in Succeeded state and approved especially for your workspace storage and any linked resources like Key Vault or SQL.

2.If your storage account has firewall rules, make sure the Synapse managed outbound IPs are allowed in that list. You can find these IPs in your workspace under Networking or Managed private endpoints.

3.Also check that public network access isn’t fully turned off for required services unless all private endpoints are properly set up.

If you need to control outbound network rules yourself (like adding NSGs or custom routes), you’d have to use a customer-managed VNet instead of a managed one that’s the only option where outbound traffic can be customized.

Please check those points and let me know how it goes. I’ll be happy to help you go through the private endpoint or firewall settings if it’s still not working.

Thanks,
Manoj
Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-10-23T08:54:45.5033333+00:00

Hi MS,

We haven't heard back from you. Please reply if you have any questions in this matter and we will gladly continue the discussion.

Thanks,
Manoj
Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-10-24T08:53:57.5633333+00:00

Hi MS,

I hope the steps provided above were helpful. Please let me know if there's anything else I can assist you with.

Thanks,
Manoj
MS 0 Reputation points

2025-10-27T17:12:47.0233333+00:00

Hi Manoj,

I've checked all Managed Private Endpoints in Synapse are in Succeeded State

All storage accounts have firewall rules but are connected to Synapse via private endpoint

as above

Kind regards, Mark
Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-10-28T10:50:25.73+00:00

Hi MS,

Based on the error you are seeing it looks like the problem might be related to how your Synapse workspace is set up, especially with the network configuration.

Here are some steps to help you fix the issue:

1.Make sure all your private endpoints (like for storage accounts or key vault) are properly set up and marked as "Succeeded." These need to be working correctly for Synapse to connect to them. Also, check the firewall settings on these services. Make sure that Synapse’s IP addresses are allowed in your storage account’s firewall.

2.Your Managed Virtual Network should be able to reach essential services like your storage and the Synapse control plane. Double-check that everything is connected properly. If you are using private endpoints, make sure that services can still communicate, even if public access is turned off for some of them.

3.Check that your Integration Runtime is working properly in Synapse Studio. If needed, try restarting or creating a new debug cluster to ensure it runs smoothly.

4.Make sure you haven’t reached any resource limits in your Synapse workspace. You can go to the "Monitor" section to check if any resources are being overused or if there are limitations.

Do let us know if you have any questions and concerns.

Thanks,
Manoj

Answer 1

Hi MS,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.
Thanks for sharing those Livy logs they are really helpful in pinpointing what’s going on. The SSL certificate error you’re seeing usually means the Synapse workspace isn’t able to securely connect to your storage account over the network.

This often happens if there’s a missing or misconfigured private endpoint for the storage account, or if the DNS isn’t set up to point Synapse to the right (private) address. Even when things look fine in the portal, these network details can cause your workspace to return errors like “certificate verify failed.”

Here’s what you can do:

Check that every storage account your workspace needs has a private endpoint set up, and it shows “Succeeded” status.

Make sure the Private DNS Zone (privatelink.blob.core.windows.net) is linked to your Synapse managed network. This helps Synapse find the right storage account over a secure, private path.

If the storage account has firewall rules, be sure the Synapse managed IPs are allowed in its settings.

If everything looks good and it’s still not working, try reconnecting the linked service and running a quick test.

Helpful References:
https://azureplayer.net/2022/12/invalidabfsrestoperationexception-in-azure-synapse-notebook/
https://stackoverflow.com/questions/74559443/azure-synapse-private-endpoints-trustfailure-ssl-tls-secure-channel
https://azureplayer.net/2022/07/mounting-adls-point-using-spark-in-azure-synapse/

Hope this helps. Do let us know if you have any further queries.

Manoj Kumar Boyini 1,165 Reputation points Microsoft External Staff Moderator

2025-11-24T21:54:37.2133333+00:00

Hi MS,

I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

Share via

Generic error when running a Synapse pipeline containing a dataflow or trying to start a debug session

1 answer

Your answer