Data Factory: Pipeline Start Delayed Without Apparent Reason — Intermittent Issue

Question

Data Factory: Pipeline Start Delayed Without Apparent Reason — Intermittent Issue

Jan 0

Problem description

I am experiencing an issue where my Azure Data Factory pipeline starts as expected, but the parallel data flow activities within the pipeline begin approximately 19 minutes later, which is unusual compared to other runs. This delay occurs despite the use of Azure Integration Runtime with TTL configured, and no explicit errors or queueing are evident in the logs.

Environment

Azure Data Factory used for an almost-daily ETL process with multiple pipelines running sequentially and some parallel. The affected pipeline has six parallel data flow activities using Azure Integration Runtime. The region, compute size, and concurrency settings are not specified in the case.

What I've already tried

I provided the start time of the affected run, a run identifier, and a description of the delay. I considered cluster cold-start as a cause but noted TTL was enabled. I reviewed self-help content about pipeline concurrency, transient conditions, and IR behavior, but no definitive cause was identified. No further diagnostic details or test results are available in the case record.

Current status

The latest run showed the pipeline starting promptly, but the six data flow branches only began after about 19 minutes. The behavior is intermittent, as other days do not show this delay. I am seeking assistance to determine whether the delay occurs during scheduling, queueing, or cluster provisioning, and how to prevent or mitigate this delay.

0 comments

1 answer

Your answer

Answer 1

Hi Jan & thx for join me at Q&A platform,

A 19-minute delay before all six parallel data flows begin is definitely worth investigating, especially if the same pipeline normally starts much faster.

The first thing I'd clarify is that having TTL enabled on Azure Integration Runtime doesn't guarantee that a warm Spark cluster will always be available. TTL allows an eligible cluster to remain available for reuse after execution, but reuse depends on the cluster configuration, available capacity, and whether the previous cluster is still within its TTL window.

If ur ETL runs almost daily, the cluster from the previous day's execution will generally have expired long before the next run. In that case, Azure Data Factory may still need to provision a new Spark cluster, even with TTL configured.

However, 19 minutes is long enough that I wouldn't automatically blame cluster cold-start. The fact that all six activities appear to start together could indicate a shared dependency, Integration Runtime capacity issue, or scheduling delay.

I'd start by comparing the pipeline's start timestamp with the individual activity run timestamps. This helps determine whether the activities were waiting to be scheduled or whether they had already started and were waiting for Spark compute.

Next, open the affected data flow activity runs in ADF Monitor and check the detailed execution information, particularly any available queueing, cluster acquisition, startup, and execution timings. Compare those with a successful run from another day.

Also check whether all six data flows use the same Azure IR configuration, including compute type and core count. Six parallel data flows don't necessarily mean six independent Spark clusters, especially when TTL-based cluster reuse is involved. Depending on the configuration and concurrency limits, activities may share available compute or wait for capacity.

I'd also look at the timing of the preceding pipelines. If they're running sequentially, confirm that the affected pipeline's dependencies actually completed when expected and that no other pipeline runs were consuming the same Integration Runtime capacity.

If the delay is happening during cluster provisioning, it could be related to transient regional compute availability or backend provisioning delays. If the activities are queued before cluster acquisition, the investigation should focus more on concurrency and scheduling.

For mitigation, u could consider running a small warm-up data flow shortly before the main ETL workload, provided the cluster configuration supports reuse. But I'd verify where those 19 minutes are being spent first, bc warm-up won't help if the real problem is activity scheduling or backend queueing.

If this happens again, I'd open an Azure Data Factory support case with the affected pipeline run ID, all six activity run IDs, UTC timestamps, IR region, compute configuration, and TTL settings. Ask the engineering team to check service-side scheduling, Spark cluster allocation, and any regional capacity delays during that specific run.

One question: do the six data flow activities show a 19-minute gap before their activity start timestamps, or do they start immediately but spend 19 minutes initializing the Spark environment?

That distinction is probably the fastest way to narrow down the root cause.

rgds,

Alex

p.s. if my answer is help - pls mark it as an answer

Jan 0 Reputation points

2026-10-26T15:00:45.95+00:00

thank you very much Alex for trying to help me with this issue. I don't see something in your suggestions that I think could help though. I added a lot of detail and screen prints to my (already opened) case but somehow the AI summarization leaves a lot out which will make it difficult for others to effectively help because it should answer a lot of your questions/suggestions.

There is a pipeline running before the problem pipeline (dependency = on completion) which leaves just the right amount of warm clusters running which can be used, also the pipeline before is completed according to logging. The dataflows just seems to do nothing for 19 minutes. It's not in cluster startup, the logging simply shows the (6) dataflows start time is 19 minutes later than the pipeline start time. There is nothing in the pipeline or dataflows that could be a dependency that is not met. So the gap is indeed before the activity start timestamp.

I set up a specially configured combination of IR's (with sufficient TTL), pipelines and dataflows (divided over IR's), which worked perfectly for a couple of weeks but since recently started behaving differently for no good reason (maybe it is the recently announced spark upgrade but of course I cannot tell)

I created a case at MS in the hope of someone looking at the backend of this and find a reason for what is happening. So far I only encountered a repetitive discussion with an agent and then the case was placed at this forum.

Not that I don’t expect anyone here to help. It could be that someone encountered something similar and has a lead on what might be happening. It is a pity though that we pay money for MS support but when you need them it really takes a lot of effort to get them to help you.