one of the pipeline is failing while saving the data back with spark context was closed

Question

one of the pipeline is failing while saving the data back with spark context was closed

Anuradha Tiwary 0

I am getting an error saying spark context was closed , it's taking an hour to run the pipeline and then it's failing because the spark context was closed . I am attaching the screenshot of the error and also the link to the logs :

https://web.azuresynapse.net/sparkui/a3b673b2-d221-464e-ab3b-d71af1479856/workspaces/synapsews-emscloud-development-central-us/sparkpools/oidreporting/livyid/29183/history/application_1780991357664_0009/1/jobs/job/?id=110

https://web.azuresynapse.net/sparkui/a3b673b2-d221-464e-ab3b-d71af1479856/api/v1/workspaces/synapsews-emscloud-development-central-us/sparkpools/oidReporting/sessions/29183/applications/application_1780991357664_0009/driverlog/stderr

screenshot :
Screenshot 2026-06-09 at 9.01.48 PM

Any help would be appreciated.

SAI JAGADEESH KUDIPUDI 3,640 Reputation points Microsoft External Staff Moderator

2026-06-09T17:18:03.5+00:00
Hi @Anuradha Tiwary ,

When a Synapse Spark pipeline fails with “Stage cancelled because SparkContext was shut down”after running for about an hour, it usually means the Spark application/session ended unexpectedly (most commonly due to resource pressure like memory/executor sizing, or a long-running task hitting a failure condition). In your case, the failure shows up at the stage that ends with save at NativeMethodAccessorImpl.java:0, and the stage failure reason explicitly says the SparkContext was shut down.

Here are the most useful things to try based on the common Synapse Spark job failure guidance:

What to check / try

Look for memory pressure and tune executor sizing

A very common cause of Spark job failures in Synapse is memory pressure (often from improper/wrong-sized executors), especially with long-running operations and expensive shuffles.

Try:

Reduce the data skew / heavy partitions (if applicable)

Increase executor memory / adjust executor cores based on the workload

Ensure you’re not creating large intermediate datasets unexpectedly

This aligns with the Synapse Spark “Spark Job failures” recommendation: memory pressure is the most common challenge, particularly when executors are wrong-sized and operations are long-running/cartesian-like.
2) Review operations that cause big shuffles / cartesian blowups

If the pipeline stage is doing joins/aggregations that can trigger large shuffles (or cartesian products), the job can fail after significant runtime.

Try:

Check the logic around joins/aggregations for unintended cartesian behavior

Optimize joins and shuffles

Consider caching only where it helps

Check whether your pipeline is doing expensive installs / session startup delays

If your pipeline (or Spark pool session) attaches custom libraries (requirements.txt / .whl, or similar) and installs a large package, that can affect startup/initialization behavior.

Try:

Confirm whether you’re attaching libraries to the pool/session for this pipeline run

If so, validate the startup time and whether it changed recently

Monitor the Spark application logs for the real root cause

The “SparkContext was shut down” message is often a downstream symptom. The actual reason is typically earlier in the driver/executor logs.

You already linked to the Spark UI job history and driver stderr—those are the right places to confirm whether the underlying error is related to memory, OOM, shuffle, or another Spark failure.

If you’re using a notebook-in-pipeline “Spark step”

If this pipeline is running a Spark notebook via a pipeline activity, ensure the input/output datasets are supported and configured correctly for that activity type (the pipeline step supporting Spark in Synapse ML has specific dataset expectations). Even though your failure is “SparkContext closed”, it’s still worth validating the pipeline activity configuration if this stage is part of a Spark step.

Quick follow-up questions (so we can narrow it down)

Can you share the actual error lines from the driver logs (a few lines above where it says SparkContext was shut down), not just the stage reason?

Are you running this on a Synapse Spark notebook inside a pipeline step, or is it a Synapse Spark job?

What Spark runtime / pool configuration are you using (Spark 3.5/3.4, executor size/number, node size, autoscale on/off)?

Does this job include any large joins, cartesian-like logic, or major shuffle/aggregation in that failing stage?

Are you attaching custom Python libraries / requirements (.txt/.whl) or other extra libraries to the session?

Microsoft References Links:

Spark job failure + common causes (memory pressure, joins/shuffles, long-running operations): https://learn.microsoft.com/azure/synapse-analytics/spark/apache-spark-performance#spark-job-failures

Monitor Spark applications (Spark UI/logs): https://learn.microsoft.com/azure/synapse-analytics/monitoring/how-to-monitor-spark-applications

Issues: starting Spark session with custom python libraries taking longer than usual: https://learn.microsoft.com/azure/synapse-analytics/known-issues? (see “Starting a Spark session (with custom python libraries) is taking longer than usual”)

Spark step dataset expectations (Synapse ML pipeline Spark step): https://docs.microsoft.com/azure/machine-learning/how-to-use-synapsesparkstep

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

2 answers

Your answer

Answer 1

Hi @SAI JAGADEESH KUDIPUDI ,

Thank you so much for your response and few things which you asked for are here :

{"TraceId":"a8e902bc-dfb6-4eaf-ad29-79c71266b544 | client-request-id : 36920f83-37e5-43ee-a465-fb3f2e837627","ErrorSource":"User","Message":"<html>\n      <head>\n        <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\"/><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"/><link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/vis-timeline-graph2d.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/webui.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/timeline-view.css\" type=\"text/css\"/><script src=\"/static/sorttable.js\"></script><script src=\"/static/jquery-3.5.1.min.js\"></script><script src=\"/static/vis-timeline-graph2d.min.js\"></script><script src=\"/static/bootstrap.bundle.min.js\"></script><script src=\"/static/initialize-tooltips.js\"></script><script src=\"/static/table.js\"></script><script src=\"/static/timeline-view.js\"></script><script src=\"/static/log-view.js\"></script><script src=\"/static/webui.js\"></script><script>setUIRoot('')</script>\n        \n        <link rel=\"shortcut icon\" href=\"/static/spark-logo-77x50px-hd.png\"></link>\n        <title>Not Found</title>\n      </head>\n      <body>\n        <div class=\"container-fluid\">\n          <div class=\"row\">\n            <div class=\"col-12\">\n              <h3 style=\"vertical-align: middle; display: inline-block;\">\n                <a style=\"text-decoration: none\" href=\"/\">\n                  <img src=\"/static/spark-logo-77x50px-hd.png\"/>\n                  <span class=\"version\" style=\"margin-right: 15px;\">3.5.1-HS-20240910.1</span>\n                </a>\n                Not Found\n              </h3>\n            </div>\n          </div>\n          <div class=\"row\">\n            <div class=\"col-12\">\n              <div class=\"row-fluid\">Application application_1781055122379_0008 not found.</div>\n            </div>\n          </div>\n        </div>\n      </body>\n    </html>\n\nStatus: 404\nResponse: \n"}

Can you share the actual error lines from the driver logs (a few lines above where it says SparkContext was shut down), not just the stage reason?

Above are few errors which I saw and other than that I just see info and exactly at 1 hour , the shutdown hook is invoked.

User's image

DRIVER LOGS :

   ---------------------------------------------------------------------------
   Py4JJavaError                             Traceback (most recent call last)
   Cell In [17], line 65
        56 if (isDebug):
        57     df_door_events.show()
        59 df_door_events\
        60     .write.format("cosmos.oltp")\
        61     .option("spark.synapse.linkedService", cosmosDBLink)\
        62     .option("spark.cosmos.container", eventContainer)\
        63     .option("spark.cosmos.write.strategy", "ItemOverwrite")\
        64     .mode('append')\
   ---> 65     .save()
   
   File /opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:966, in DataFrameWriter.save(self, path, format, mode, partitionBy, **options)
       964     self.format(format)
       965 if path is None:
   --> 966     self._jwrite.save()
       967 else:
       968     self._jwrite.save(path)
   
   File ~/cluster-env/env/lib/python3.10/site-packages/py4j/java_gateway.py:1321, in JavaMember.__call__(self, *args)
      1315 command = proto.CALL_COMMAND_NAME +\
      1316     self.command_header +\
      1317     args_command +\
      1318     proto.END_COMMAND_PART
      1320 answer = self.gateway_client.send_command(command)
   -> 1321 return_value = get_return_value(
      1322     answer, self.gateway_client, self.target_id, self.name)
      1324 for temp_arg in temp_args:
      1325     temp_arg._detach()
   
   File /opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py:190, in capture_sql_exception.<locals>.deco(*a, **kw)
       188 def deco(*a: Any, **kw: Any) -> Any:
       189     try:
   --> 190         return f(*a, **kw)
       191     except Py4JJavaError as e:
       192         converted = convert_exception(e.java_exception)
   
   File ~/cluster-env/env/lib/python3.10/site-packages/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
       324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
       325 if answer[1] == REFERENCE_TYPE:
   --> 326     raise Py4JJavaError(
       327         "An error occurred while calling {0}{1}{2}.\n".
       328         format(target_id, ".", name), value)
       329 else:
       330     raise Py4JError(
       331         "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
       332         format(target_id, ".", name, value))
   
   Py4JJavaError: An error occurred while calling o4549.save.
   : org.apache.spark.SparkException: Job 106 cancelled because SparkContext was shut down
   	at org.apache.spark.scheduler.DAGScheduler.$anonfun$cleanUpAfterSchedulerStop$1(DAGScheduler.scala:1196)
   	at org.apache.spark.scheduler.DAGScheduler.$anonfun$cleanUpAfterSchedulerStop$1$adapted(DAGScheduler.scala:1194)
   	at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
   	at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:1194)
   	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:2897)
   	at org.apache.spark.util.EventLoop.stop(EventLoop.scala:84)
   	at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:2794)
   	at org.apache.spark.SparkContext.$anonfun$stop$12(SparkContext.scala:2285)
   	at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1484)
   	at org.apache.spark.SparkContext.stop(SparkContext.scala:2285)
   	at org.apache.spark.api.java.JavaSparkContext.stop(JavaSparkContext.scala:550)
   	at org.apache.livy.rsc.driver.SparkEntries.stop(SparkEntries.java:142)
   	at org.apache.livy.repl.AbstractSparkInterpreter.close(AbstractSparkInterpreter.scala:195)
   	at org.apache.livy.repl.SparkInterpreter.close(SparkInterpreter.scala:152)
   	at org.apache.livy.repl.Session.$anonfun$close$1(Session.scala:692)
   	at org.apache.livy.repl.Session.$anonfun$close$1$adapted(Session.scala:690)
   	at scala.collection.mutable.HashMap$$anon$2.$anonfun$foreach$3(HashMap.scala:158)
   	at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
   	at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
   	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
   	at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:158)
   	at org.apache.livy.repl.Session.close(Session.scala:690)
   	at org.apache.livy.repl.ActiveSessions.$anonfun$closeAndRemoveAllSessions$1(ActiveSessions.scala:174)
   	at org.apache.livy.repl.ActiveSessions.$anonfun$closeAndRemoveAllSessions$1$adapted(ActiveSessions.scala:172)
   	at scala.collection.mutable.HashMap$$anon$2.$anonfun$foreach$3(HashMap.scala:158)
   	at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
   	at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
   	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
   	at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:158)
   	at org.apache.livy.repl.ActiveSessions.closeAndRemoveAllSessions(ActiveSessions.scala:172)
   	at org.apache.livy.repl.ReplDriver.preShutdownHook(ReplDriver.scala:108)
   	at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:131)
   	at org.apache.livy.rsc.driver.RSCDriver.handle(RSCDriver.java:454)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.apache.livy.rsc.rpc.Rpc.handleCall(Rpc.java:322)
   	at org.apache.livy.rsc.rpc.Rpc.handleMsg(Rpc.java:271)
   	at org.apache.livy.rsc.rpc.RpcDispatcher.channelRead0(RpcDispatcher.java:75)
   	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
   	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
   	at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
   	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
   	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
   	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
   	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
   	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
   	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
   	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
   	at java.lang.Thread.run(Thread.java:750)
   	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:958)
   	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2418)
   	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2439)
   	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2458)
   	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2483)
   	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1028)
   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
   	at org.apache.spark.rdd.RDD.withScope(RDD.scala:407)
   	at org.apache.spark.rdd.RDD.collect(RDD.scala:1027)
   	at org.apache.spark.RangePartitioner$.sketch(Partitioner.scala:304)
   	at org.apache.spark.RangePartitioner.<init>(Partitioner.scala:171)
   	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.prepareShuffleDependency(ShuffleExchangeExec.scala:293)
   	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.shuffleDependency$lzycompute(ShuffleExchangeExec.scala:173)
   	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.shuffleDependency(ShuffleExchangeExec.scala:167)
   	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.mapOutputStatisticsFuture$lzycompute(ShuffleExchangeExec.scala:143)
   	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.mapOutputStatisticsFuture(ShuffleExchangeExec.scala:139)
   	at org.apache.spark.sql.execution.exchange.ShuffleExchangeLike.$anonfun$submitShuffleJob$1(ShuffleExchangeExec.scala:68)
   	at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:268)
   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
   	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:265)
   	at org.apache.spark.sql.execution.exchange.ShuffleExchangeLike.submitShuffleJob(ShuffleExchangeExec.scala:68)
   	at org.apache.spark.sql.execution.exchange.ShuffleExchangeLike.submitShuffleJob$(ShuffleExchangeExec.scala:67)
   	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.submitShuffleJob(ShuffleExchangeExec.scala:115)
   	at org.apache.spark.sql.execution.adaptive.ShuffleQueryStageExec.shuffleFuture$lzycompute(QueryStageExec.scala:174)
   	at org.apache.spark.sql.execution.adaptive.ShuffleQueryStageExec.shuffleFuture(QueryStageExec.scala:174)
   	at org.apache.spark.sql.execution.adaptive.ShuffleQueryStageExec.doMaterialize(QueryStageExec.scala:176)
   	at org.apache.spark.sql.execution.adaptive.QueryStageExec.materialize(QueryStageExec.scala:82)
   	at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$6(AdaptiveSparkPlanExec.scala:272)
   	at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$6$adapted(AdaptiveSparkPlanExec.scala:270)
   	at scala.collection.Iterator.foreach(Iterator.scala:943)
   	at scala.collection.Iterator.foreach$(Iterator.scala:943)
   	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
   	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
   	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
   	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
   	at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:270)
   	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
   	at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:241)
   	at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:384)
   	at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:369)
   	at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:230)
   	at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:268)
   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
   	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:265)
   	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:226)
   	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:364)
   	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2$(WriteToDataSourceV2Exec.scala:362)
   	at org.apache.spark.sql.execution.datasources.v2.AppendDataExec.writeWithV2(WriteToDataSourceV2Exec.scala:253)
   	at org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec.run(WriteToDataSourceV2Exec.scala:341)
   	at org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec.run$(WriteToDataSourceV2Exec.scala:340)
   	at org.apache.spark.sql.execution.datasources.v2.AppendDataExec.run(WriteToDataSourceV2Exec.scala:253)
   	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
   	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
   	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
   	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:152)
   	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:111)
   	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:183)
   	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:97)
   	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
   	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:66)
   	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:152)
   	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:145)
   	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
   	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
   	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
   	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
   	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
   	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
   	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
   	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
   	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
   	at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:145)
   	at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:129)
   	at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:123)
   	at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:183)
   	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:901)
   	at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:330)
   	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:249)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
   	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
   	at py4j.Gateway.invoke(Gateway.java:282)
   	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
   	at py4j.commands.CallCommand.execute(CallCommand.java:79)
   	at py4j.GatewayConnection.run(GatewayConnection.java:238)
   	at java.lang.Thread.run(Thread.java:750)
   
   
   End of LogType:stdout
   ***********************************************************************

Are you running this on a Synapse Spark notebook inside a pipeline step, or is it a Synapse Spark job?

It's a schedule pipeline and it's triggered everyday as per the schedule.

What Spark runtime / pool configuration are you using (Spark 3.5/3.4, executor size/number, node size, autoscale on/off)?

Configuration are as below in the screenshot :
User's image

Does this job include any large joins, cartesian-like logic, or major shuffle/aggregation in that failing stage? There are aggregations , but they work fine in higher environments like QA . It's failing for DEV env .
Are you attaching custom Python libraries / requirements (.txt/.whl) or other extra libraries to the session?
No , not as part of the notebook.

I hope above information will be enough to investigate the failure reason . Please let me know if you need any further specific info.

Your help is much appreciated.

SAI JAGADEESH KUDIPUDI 3,640 Reputation points Microsoft External Staff Moderator

2026-06-10T21:09:00.5833333+00:00

@Anuradha Tiwary Thanks for Providing Requested Details
I understand your concern, and I have noted the information. I will check and get back to you.

Answer 2

The failure message “Stage cancelled because SparkContext was shut down” means the Spark application (driver or executors) terminated before the final write stage completed. In Synapse this is usually caused by an earlier failure (often memory‑related) that kills the session; the last stage then only reports that the context is already closed.

Use the Spark monitoring and logs to find the real root cause earlier in the run:

Open the failed run in Synapse Studio and go to the application detail page.
Check the Properties panel for the primary error message and failing stage.
Go to the Logs tab and inspect Driver logs. Search for ERROR, Exception, OutOfMemoryError, or similar just before the time the context was closed.
From the Jobs tab, expand the failed job and open the Spark UI for the failing stage. Look at the task‑level error and which executor failed.
If the logs show driver or executor OOM, data skew, or schema issues, fix those (for example, reduce data volume per partition, avoid wide collect()/toPandas() on the driver, or correct schema mismatches) and rerun.

If the logs do not show a clear application error and the context ends abruptly, also check for infrastructure issues such as cluster resize, spot node loss, or capacity problems, using the monitoring hub and capacity metrics.

Once the underlying error is resolved, the final write stage should complete without the “SparkContext was shut down” cancellation.

References: