Azure Synapse Spark Pool: Executor added mid-execution fails to acquire token, causing job failure

Hello Roman Shamardin,

Greetings! Thanks for raising this question in Q&A forum.

You have provided an excellent and detailed error report. The key signal is in this part of the error message:

Message=Entity SparkV35-987 not found
Error Component: LSR

The LSR (Linked Service Resolver) component is the internal Synapse service that resolves pool identity context when executors request storage tokens. The error Entity SparkV35-987 not found means that when the replacement executor was provisioned mid-execution and attempted to call the Token Service to acquire a storage access token, the LSR could not find the pool's registration entry in its backend lookup. This is a race condition between the executor provisioning timeline and the LSR's pool entity registration.

Here is what is happening and what you can do.

Root cause explanation:

When a long-running Spark job triggers executor replacement (due to node failure, spot eviction, or autoscale-triggered node replacement), the new executor node needs to register itself with the Synapse Token Service (which uses LSR internally) to prove it is an authorized member of pool SparkV35-987 before it can request storage tokens. In some cases, particularly for long-running jobs where the initial pool session has been running for many hours, there is a timing window where the replacement executor comes online before its pool entity context has been fully propagated or refreshed in LSR. The Entity SparkV35-987 not found message is the LSR saying it cannot find the pool registration entry to validate the token request against.

This is a known scenario that Microsoft is aware of for Spark 3.x pools with autoscale or dynamic executor allocation enabled for long-running jobs.

Step 1: Open an Azure Support ticket immediately with your trace IDs.

This specific class of error (Entity SparkPool not found from the LSR) requires a backend investigation against the LSR service logs for your workspace. The trace IDs you have already captured are exactly what the support team needs. Please open a support ticket immediately and include:

• Workspace name: hasonorthcentralus75c63383c7cd32ae1209
• Spark Pool name: SparkV35-987
• Application ID: application_1782261914206_0001
• First Trace ID: 249853a3-7ea9-4e79-95f4-b7e4813aa950
• Client Request ID from the error: 704bbf75-2d25-4f93-afcc-1e037219cad8
• Error Component: LSR
• State explicitly: "Replacement executor added mid-execution cannot acquire Token Service registration. Entity SparkV35-987 not found in LSR. Requesting investigation of LSR registration state for this pool."

Go to Azure portal > Help + Support > New support request > Issue type: Technical > Service: Azure Synapse Analytics.

Step 2: As a short-term mitigation, reduce reliance on executor replacement by adjusting your autoscale and dynamic allocation settings.

The failure happens because a replacement executor is added mid-execution. You can reduce how aggressively new executors are brought in by setting more conservative dynamic allocation bounds in your Spark configuration. Add this to your Spark job or notebook configuration:

%%configure -f
{
  "conf": {
    "spark.dynamicAllocation.enabled": "true",
    "spark.dynamicAllocation.minExecutors": "2",
    "spark.dynamicAllocation.maxExecutors": "10",
    "spark.dynamicAllocation.executorIdleTimeout": "300s",
    "spark.dynamicAllocation.cachedExecutorIdleTimeout": "600s",
    "spark.task.maxFailures": "8"
  }
}

Increasing spark.task.maxFailures above the default of 4 gives failing tasks more retries during which the executor registration may complete, which can allow the job to recover without full failure. Setting a longer executorIdleTimeout reduces the frequency of executor churn that triggers replacement.

Step 3: Consider switching storage access to use the workspace Managed Identity directly instead of the LSR token path.

The stack trace shows the error is occurring in VStreamHttpUtils calling through TokenLibraryUtils.getfsAccessToken, which routes through LSR. For long-running jobs that read from ADLS Gen2 primary storage linked to the workspace, you can configure Spark to use the workspace Managed Identity directly with OAuth, bypassing the Synapse Token Service entirely:

spark.conf.set(
    "fs.azure.account.auth.type.<your-storage-account>.dfs.core.windows.net",
    "OAuth"
)
spark.conf.set(
    "fs.azure.account.oauth.provider.type.<your-storage-account>.dfs.core.windows.net",
    "org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider"
)
spark.conf.set(
    "fs.azure.account.oauth2.msi.endpoint.<your-storage-account>.dfs.core.windows.net",
    "http://169.254.169.254/oauth2/token"
)

This uses IMDS (Instance Metadata Service) to get a Managed Identity token directly on each executor node, which does not depend on the LSR pool entity registration and therefore will not be affected by the mid-execution replacement timing issue.

Step 4: Verify your Spark Pool autoscale configuration is correctly specified.

Ensure that your pool's autoscale min and max node counts are explicitly set and that dynamic executor allocation has explicit min and max executor values configured at the pool level. In scenarios where the Dynamic Allocation option is enabled in a Synapse Spark Pool, the platform reserves the number of executors based on the maximum limit specified by the user for any Spark application submitted. If these are not set correctly, the platform may make suboptimal scaling decisions that increase executor replacement frequency.

You can verify or update via Azure CLI:

az synapse spark pool update \
  --name SparkV35-987 \
  --workspace-name hasonorthcentralus75c63383c7cd32ae1209 \
  --resource-group <your-resource-group> \
  --enable-dynamic-exec \
  --min-executors 3 \
  --max-executors 20

If this answer helps you kindly accept the answer which will help others who have similar questions.

Best Regards,

Jerald Felix.