Data Sync Delays with Microsoft Synapse

@Steve Busi

Thank you for sharing your experience with the data export process from Dynamics to the Data Lake via Dataverse. I understand how critical timely data synchronization is, especially during month-end reporting.

Even with upgrades like increasing the number of Spark pool cores, longer sync times can still occur due to several contributing factors. Since the new pipeline involves exporting data via Dataverse and Synapse Link, and converting it to Parquet format, there’s inherently more structure and transformation involved compared to the previous “Export to Data Lake” approach. That added complexity often comes with trade-offs in performance.

Here are a few areas worth exploring to help reduce the delay -

Spark Job Performance Tuning

Analyze job stages and task distribution to identify slowdowns, common culprits include skewed partitions or large shuffles.

Consider reviewing the number of executor cores and memory allocations; sometimes increasing parallelism at the task level yields more benefit than just bumping overall core count.

Data Partitioning Strategy

Ensure proper partitioning during Parquet write operations. Partitioning by high-cardinality columns (e.g., timestamp) can greatly improve performance and write parallelism.

If you are writing to Delta Lake, consider tuning Z-order or OPTIMIZE (for downstream reads).

Storage Throughput & Configuration

Verify your ADLS Gen2 account's performance tier, standard tiers can become bottlenecks with large volumes.

Also, confirm if hierarchical namespace is enabled and being used optimally for folder/file operations.

Sync Frequency & Batching

Instead of syncing large datasets in one go, splitting the sync into smaller, more frequent incremental jobs can help reduce lag and resource contention.

Codec & File Write Optimization

Using efficient compression codecs (e.g., Snappy) and tuning row group size during Parquet file creation can shave off a notable amount of time in write operations.

Given the time sensitivity of your scenario, especially during reporting periods, one potential workaround is to temporarily run a parallel export for high-priority tables using the legacy “Export to Data Lake” approach while continuing to optimize the new Synapse Link setup for broader use cases.

Also worth noting - Microsoft’s own guidance for Synapse Link with Dataverse notes that latency can vary from a few minutes to over an hour, depending on data change volume and pipeline configuration, so there may be some inherent trade-offs with the real-time expectation.

For additional information, please refer: https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-transition-from-fno#known-issues-and-workarounds

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

@Steve Busi

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

I'll check with the team. Thanks.

@Steve Busi

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

@Steve Busi
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.