12,207 questions with Azure Data Factory tags

Sort by: Updated
2 answers

MS Fabric// Unable to connect with Lakehouse with Data gen2.

Unable to add the Lakehouse as Destination to DATAGen2 getting this error message. What can be the reason

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-22T06:45:43.2333333+00:00
Verma, Namrata 0 Reputation points
edited an answer 2026-10-03T12:38:05.65+00:00
CrazyKats 19,920 Reputation points Volunteer Moderator
1 answer

ADF Copy Activity Issue - Copy sql data -> ADLS -> Snowflake

Hi, We got a data flow with Azure managed sql instance db as source. CDC is enabled (SQL Server CDC) in the source with Allow schema drift and Infer drifted column types. Sink is the ADLS gen 2 type as dataset and allow schema drift. There is derived…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-10-02T10:23:39.63+00:00
Anil Kestur 5 Reputation points
answered 2026-10-02T11:16:04.46+00:00
Senthil kumar 2,580 Reputation points
0 answers

How do you handle incremental loads in Azure Data Factory when different sources have different change patterns?

When building an Azure data platform, scheduled loads are fairly straightforward. Incremental loads can be more complicated when each source handles changes differently. For example, some systems may have a reliable timestamp or change indicator, while…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-10-01T14:56:52.1666667+00:00
Kavika Roy 0 Reputation points
1 answer

How do you handle API ingestion in Azure Data Factory when the source system is an ERP?

A common pattern with ERP integrations is to use Azure Data Factory as the orchestration layer and APIs as the source interface, rather than making changes to the ERP itself. One practical architecture is: ERP/API → Azure Data Factory → ADLS Gen2 →…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-10-01T11:42:13.6033333+00:00
solanki srikanth 0 Reputation points
answered 2026-10-01T12:10:32.77+00:00
Senthil kumar 2,580 Reputation points
1 answer

How to Ingest a table of xml data from on-prem DB2 warehouse to ASDLGen2 using ADF

Hi, I am using ADF to incrementally load a table containing 2 XML columns from on-prem DB2 warehouse to ADSLGen2 in parquet format. In Copy activity, I use DB2 connector at source and parquet dataset at sink; I also use Query with XMLSERIALIZE for these…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-16T14:13:37.9933333+00:00
Nhan, Phuong 0 Reputation points
commented 2026-09-30T17:17:09.08+00:00
Himaja Y 555 Reputation points Microsoft External Staff Moderator
2 answers

Issues Azure data Factory SFTP connection ( SSH client)

HI MS Team, We encounter some issues with connecting a SFTP server of one of our financial counter parties. The issue is due to security level that they do not support rsa-sha  but only the rsa-sha2  version for host key exchange. If i look at the error…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-07T10:30:11.16+00:00
Shivata Tikoo 0 Reputation points
commented 2026-09-30T01:05:32.9133333+00:00
SIVASANKAR YEDDULA 250 Reputation points Microsoft External Staff Moderator
1 answer

Restrict Non-Owners from changing the Collaboration branch of Azure Data Factory using Azure Policy

We would like to impose the following restrictions on Azure Data Factory. We were able to achieve the #1 feature using the below policy { "mode": "All", "policyRule": { "if": { …

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2023-02-22T05:51:14.7966667+00:00
Praveen Sreeram 11 Reputation points
edited a comment 2026-09-28T11:35:32.23+00:00
JuliaMarvin 23,005 Reputation points Volunteer Moderator
2 answers

Trying to filter csv file based on content

I have an existing ADF pipeline that is creating an error due to some bad (or not acceptable data) displayed on the csv file. Basically, instead of having proper header on the first row with certain amount columns (with data), csv file displays data like…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-07-22T21:45:14.68+00:00
Justin Doh 1,000 Reputation points
commented 2026-09-26T01:45:13.7933333+00:00
Justin Doh 1,000 Reputation points
1 answer

How can I implement incremental API ingestion from Odoo ERP to Azure Data Lake Storage Gen2 using Azure Data Factory?

I'm working for a project on an Azure data platform where Odoo ERP is the source and ADLS Gen2 is used as the data lake. The Odoo API contains manufacturing, inventory, procurement, and financial data that needs to be refreshed regularly for Power BI…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-25T14:43:28.43+00:00
Kavika Roy 0 Reputation points
answered 2026-09-25T15:01:47.1033333+00:00
Allan Solomon Mejia 10,225 Reputation points
2 answers One of the answers was accepted by the question author.

How can we optimize ADF pipelines that uses control tables and lookup activities

Hi There , I am using ADF pipelines for data migration from on prem to azure , i had been doing it from a long and the thing i face most of the time is when we use control table and look up activities , the pipelines became slow . I just want to know…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-21T11:59:55.3033333+00:00
Aditya Singh Rathore 195 Reputation points
commented 2026-09-25T12:32:40.5966667+00:00
Maraiah Kasetti (Wipro Limited) 85 Reputation points Microsoft External Staff Moderator
0 answers

Data Factory: HubSpot Connector Out of Date — Errors with Certain Endpoints

Problem description I am experiencing errors when using the Azure Data Factory HubSpot connector 2.0, particularly when accessing certain endpoints like the marketing campaigns table during a source-data update pipeline. The behavior did not meet my…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-16T23:15:08.8+00:00
Noah Daniels 0 Reputation points
commented 2026-09-23T17:33:34.2033333+00:00
Noah Daniels 0 Reputation points
2 answers

Error occurring during the SSH handshake with ADF

I am trying to find a solution on this error when I tried to copy data from a client who uses AWS for SFPT. I am using ADF (Azure Data Factory) to move data from this client's SFTP. Bottom is details of error…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-14T21:28:00.8833333+00:00
Justin Doh 1,000 Reputation points
commented 2026-09-21T22:54:35.98+00:00
Justin Doh 1,000 Reputation points
2 answers

Unable to delete data workflows gen2 tasks under fabric workspace

Hi Team, Under fabric workspace, we have created Gen2 dataflow tasks. We would like to delete the Gen2 Workflow tasks, by removing the linages, but we are getting error. Then same is able to delete after some days, either it takes 1 week or 10days. In…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-18T10:37:53.7866667+00:00
Santosh Kumar Joshi 0 Reputation points
answered 2026-09-21T15:23:14.43+00:00
Josh Turnbull 20 Reputation points
2 answers

Update of parentaccountid into CRM from ADF

I'm trying to update records from a .csv file through ADF copy activity into CRM. It's working fine for the GUIDs present for parentaccountid and updating accordingly. However, when a blank value is sent for this field, it's not removing the existing…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-16T10:54:54.49+00:00
Vidyashree Dinesh 0 Reputation points
edited a comment 2026-09-21T12:22:23.38+00:00
Stijn De Vos 0 Reputation points
1 answer

Azure Data Factory Copy Activity fails extracting ZIP files to ADLS Gen2 with BadRequest for Unicode/Vietnamese filenames — possible ZIP filename encoding issue

We are using Azure Data Factory Copy Activity to extract ZIP files and write the extracted files to Azure Data Lake Storage Gen2. The Copy Activity fails for some files/directories inside the ZIP with the following error: The ZIP contains…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-17T04:19:23.85+00:00
Balasuriya R 0 Reputation points
answered 2026-09-21T11:54:39.9+00:00
Aditya Singh Rathore 195 Reputation points
1 answer

How to bypass locked temporary Office files (~$*) during bulk FileServer-to-ADLS copy in ADF without high cost?

Hi Everyone, I am building an incremental ingestion pipeline in Azure Data Factory (ADF) to copy ~70,000 files (across 47 different file types) from an on-premises FileServer into ADLS Gen2 using a Self-Hosted Integration Runtime (SHIR). The Challenge We…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-18T14:45:46.0133333+00:00
Misbah Bin Hossain 0 Reputation points
edited the question 2026-09-21T07:07:05.01+00:00
Misbah Bin Hossain 0 Reputation points
4 answers

Copy data from Sharepoint using Authorization: Bearer token getting status code 401 Unauthorized

ErrorCode=HttpRequestFailedWithUnauthorizedError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http request failed with status code 401 Unauthorized, usually this is caused by invalid credentials, please check your activity…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2026-09-03T11:20:09.3133333+00:00
Prakash Palanivelu 0 Reputation points
answered 2026-09-20T06:43:09.7066667+00:00
Smaran Thoomu 35,870 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

Setup ADF authen with personal github

I have error when I try setup repo git integration with new credential it's not working. check in tab network it return 401 but I don't see any step input credential or token???? why it have token in here?? where it come from??

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2024-05-10T05:21:26.4366667+00:00
Duy Nguyen 20 Reputation points
commented 2026-09-17T12:11:24.4233333+00:00
Janne Kujanpää 266 Reputation points
2 answers One of the answers was accepted by the question author.

Recommended technologies for creating a dashboard of ongoing jobs.

I'm looking for recommendations on technologies to use for the following project. I need to create a dashboard of summarized job data from a network of hundreds of users. Our current application is used in the field for inputting and tracking job costs…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2024-05-09T14:00:50.17+00:00
COW Farmer 45 Reputation points
accepted 2026-09-16T15:54:20.8266667+00:00
COW Farmer 45 Reputation points
1 answer

Ingest multiple xml files through ADF which has different schemas

Hi all, I have an Azure blob storage of xml files with at least two different schemas. I would like to ingest all of these files through ADF (probably using Dataflow) and only process one type of schema to sink to a SQL database. How is this done…

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

12,207 questions
asked 2023-05-08T15:25:06.67+00:00
Raymond Mitchell 15 Reputation points
commented 2026-09-16T13:34:05.1933333+00:00
Emilio Molina 0 Reputation points